Wednesday, 2012-08-29

*** perkiset (3fe224a9@gateway/web/freenode/ip.63.226.36.169) has joined #wikid00:18
perkisetGreets everyone - is Nick about?00:19
perkisetPing ...00:47
joevanoperkiset: he usually shows up around 9:00 AM EDT most days... sometimes a little later12:02
joevanoin abaut an hour12:03
*** nowen (~nowen@99.174.93.102) has joined #wikid13:16
nowenmorning13:16
nowenperkiset: howdy13:17
Guest43741hi nick14:23
Guest43741ready to try again14:23
nowenyes14:23
Guest43741looks like the working server is also caught in a loop14:23
Guest43741and ran out of memory14:24
nowenhmm14:24
nowenthat's not good14:24
Guest43741rebooting14:26
nowenI think we should update the slave server and then try to start it as the new master14:26
nowencan you take a snapshot?14:26
Guest43741yes14:27
Guest43741should i email it to you14:27
nowenno - I mean can you take a snapshot of the VM so we can revert if needed14:27
Guest43741yes14:28
Guest43741done14:28
Guest43741however i think the damage is done14:28
Guest43741i think we promted the new server to a master and everything broke14:30
Guest43741can i bring up the working server as master14:30
nowenyes14:30
Guest43741and demote new server to slave14:30
nowenbring up the working server and rerun set up.  set replication to none14:31
Guest43741looks like both broke14:33
Guest43741caught in an endless loop14:33
nowenoj14:33
nowenok14:33
Guest43741my have to rebuild new one and restore old one14:34
nowenon the slave, run 'wget http://wikidsystems-dl.com/wikid-server-enterprise-3.5.0.b1330-1.noarch.rpm'14:34
nowenmake a backup14:34
Guest43741both broken why make bakcup14:34
nowenrun 'wget http://wikidsystems-dl.com/wikid-utilities-3.4.2-1.i386.rpm'14:34
nowenjust do it14:34
*** cmatthews (4404e717@gateway/web/freenode/ip.68.4.231.23) has joined #wikid14:34
cmatthewsOwenn you about?14:34
nowenso we can revert if need14:34
nowencmatthews: yes14:35
cmatthewscool14:35
cmatthewsThis is Clark from EPL14:35
cmatthewsOur primary token server is not responding to token requests or the web page14:35
nowenHey!14:35
cmatthewswe are able to login as root14:35
nowenis there disk space?14:36
cmatthewswhat do you think the firast step shoudl be? bad restart of the system last night14:36
nowenrun 'df -h'14:36
nowenand check to make sure there is disk space14:36
nowencmatthews: and 'netstat -anp | grep 443'14:37
Guest43741ran wget on the slave14:37
nowenGuest43741: after you have both files, run 'rpm -Uvh wikid-*' to upgrade the rpms14:38
cmatthewsplenty of space running the netstat now14:39
cmatthewsrelaying hehe... I'm out of the office14:39
Guest43741i keep getting out of memroy14:39
cmatthewsunable to connect without the two factor14:39
nowenGuest43741: how much memory is in the vm?14:40
nowencmatthews: awkward..14:40
Guest437411024 but its a run away processes14:40
Guest43741thats causing the issues14:40
cmatthewsnetstat returned nothing14:40
cmatthewsjust #14:41
nowenGuest43741: run 'top' and then shift-M  the memory hog will be on top14:41
nowencmatthews: what happens when you run 'wikidctl start'?  do you get an error14:41
cmatthewsthat did it... thanks bro.14:43
nowenhehe, so you needed to start it? ;)14:44
cmatthewsSupport powered the environment down last night ... been literally months and months maybe ever 18 months since we had that box restart,14:44
cmatthewssolid as a rock14:44
cmatthewsnot in the office14:44
cmatthewsnot in front of my materials14:44
cmatthewsso you saved us thanks14:44
nowencmatthews: also, if you upgrade, there is a  service script you can set up14:44
nowento start automagically14:44
nowenI'm going to do a new release later today though, so wait for that14:45
nowenor you can get it now:  'wget http://wikidsystems-dl.com/wikid-server-enterprise-3.5.0.b1330-1.noarch.rpm'14:45
nowenand wget http://wikidsystems-dl.com/wikid-utilities-3.4.2-1.i386.rpm'14:45
nowenGuest43741: what is chewing up the memory?14:46
cmatthewsSounds like a plan, probably 2-3 weeks, a lot changing and don't need another variable of disaster. Thanks for the links, saved.14:46
nowenI understand ;)14:46
nowenif it ain't broke14:46
Guest43741system slow to respond14:46
nowenbut it would avoid another morning like todays14:46
cmatthewsNah, good stuff would like to update. Just whole VM environment upgrading atm.. ehhe14:47
nowenGuest43741: run 'killall -9 java'14:47
Guest43741doesn't do anything14:47
Guest43741might as well reinstall on top of it14:47
nowenwhat did top report?14:47
nowenbecause you can easily kill any process14:48
nowenand reinstalling and re-syncing doesn't seem easy to me14:48
nowenso please run top14:50
Guest43741doesn't do anything14:50
Guest43741top -M14:50
Guest43741i ran but nothing14:50
nowenjust 'top'14:50
nowenthen when you see the list of processes, hit capital M14:51
cmatthewsttyl Nick, thanks again for quick solve. Product is strong and serves it's purpose nicely. Keep up the good work..14:51
nowenthanks, cmatthews!14:51
Guest43741i see procesess14:52
Guest43741it appears to be shutting down process14:52
Guest43741i think reinstalling is fater way to go though14:52
nowenGuest43741: no, it is not14:52
nowenwhen you hit shift-M what is the top process?14:53
Guest43741its rebooting automatically14:53
Guest43741the working server is no longer working either14:54
Guest43741anyway the new server is back to logon screen14:54
nowenok - log in.  the run away process should be dead14:54
Guest43741just logged in14:54
nowenrun 'rpm -Uvh wikid-*'14:54
Guest43741done14:55
*** cmatthews has quit (Ping timeout: 245 seconds)14:55
Guest43741should i run this on both machines14:56
nowenno, just the new server14:57
Guest43741should i try the update on the new server14:57
nowenyes - we are only working on the new server14:58
Guest43741ok14:58
Guest43741its downloading update14:59
Guest43741brb14:59
Guest43741finished15:05
Guest43741downloading15:06
Guest43741hello15:09
Guest43741still there15:10
nowendid you run the rpm command?15:13
Guest43741ran that before update15:14
Guest43741should i run it again15:15
nowenno15:15
nowenrun 'rpm -qa | grep wikid'15:15
nowenwhat does it say>15:15
Guest43741'wikid-utilities-3.4.2.-115:16
Guest43741wikid-server-enterprise-3.4.87.b1030-115:16
Guest43741wikid-appliance-3.4.21.b126-115:16
Guest43741thats all15:16
nowenthat's not the right server rpm15:16
nowenplease run  'wget http://wikidsystems-dl.com/wikid-server-enterprise-3.5.0.b1330-1.noarch.rpm'15:17
Guest43741ran that already15:18
Guest43741i think we need to unpack it or install it15:18
nowenrun 'rpm -Uvh wikid-server-enterprise-3.5.0.b1330-1.noarch.rpm'15:19
nowenalso, go ahead and run 'wget http://wikidsystems-dl.com/wikid-appliance-3.5.0.b1342-1.noarch.rpm'15:19
nowenand 'rpm -Uvh wikid-appliance-3.5.0.b1342-1.noarch.rpm'15:20
Guest43741runaway processes are back15:23
Guest43741its trying to stop processes and it says xxx process already stopped15:24
Guest43741and same processes keep repeating15:24
nowenwhat process?15:24
Guest43741timecop, tomcat, database, logger15:25
nowendid you run wikidctl start?15:25
Guest43741nope15:25
nowenwhy is it starting?15:25
Guest43741don't know just ran the command as per ur instructions above15:26
nowenhmm15:26
nowenok - run 'killall -9 java'15:26
nowenand then, 'rpm -qa | grep wikid'15:26
Guest43741can't15:27
Guest43741its looping15:27
nowentry ctrl-c15:28
nowenor ctrl-z15:28
Guest43741doesnt' work15:28
Guest43741nothing works15:29
Guest43741may have to poweroff15:29
nowenok - if you want to re-install, all the users will have to re-register15:29
Guest43741no i have working copy (backup of vhd)15:29
Guest43741what I will do is restore that15:30
nowenok - try again - reboot the server15:30
Guest43741then create a new vm reinstall wikid15:30
Guest43741and start replication15:30
nowenthen then, 'rpm -qa | grep wikid'15:30
Guest43741because both servers are broken now15:30
nowenthat is what we did last time and it didn't work - why will it work this time?15:30
Guest43741this time upgrade to the latest versin then try???... maybe somethign corrupt last time... i don't know15:32
nowenthat might work15:32
Guest43741ok i will try now15:33
*** nowen has quit (Quit: Leaving.)15:42
*** nowen (~nowen@99-174-93-102.lightspeed.tukrga.sbcglobal.net) has joined #wikid15:47
Guest43741both ready now15:52
Guest43741should i upgrade the new server15:52
nowenso - both servers are running the latest wikid rpms?15:52
Guest43741no15:52
Guest43741the working server i restored to the working version...  this one is in as if nothing changed15:53
Guest43741the new server started from scratch15:53
nowenI suggest we try to upgrade WiKID on a clone of the working server15:53
Guest43741i'm going to run setup and configure network access15:53
Guest43741then run updates15:53
Guest43741then maybe run upgrade15:54
nowenno, I think there is a bug with the replication in the version you are using15:54
nowenquestion:  when you say "working server" can you reboot it or does it have the kernel issue?15:54
Guest43741i can reboot it... no kernel issue on this one... however this one is old (origianl)... setup last year15:55
nowenI see15:55
nowenand it is not set up for replication?15:55
Guest43741Nothing is at this point15:55
Guest43741i started from scratch15:56
nowenbut it has your users?15:56
Guest43741yes15:57
Guest43741its got everything15:57
nowenok15:58
Guest43741as i restored the VHD file from bakcup15:58
nowenso, yes, then go ahead and run 'yum update'15:58
Guest43741running yum update now16:03
Guest43741ok ready to retry16:25
Guest43741just ran yum update16:25
nowenok16:25
Guest43741should we try the upgrade16:26
Guest43741first16:27
nowenjust try to reboot16:29
Guest43741rebooted and logged in16:32
nowenok - try to start the server16:34
Guest43741wikidctl start ?16:36
nowenyes16:36
Guest43741i get after starting a few service... 'the database is not yet configured for WiKID. Pleae run /opt/wikid/sbin/load-db.sh16:37
nowen check the WiKIDAdmin to see if the users are there16:38
Guest43741can't login16:43
nowenso, that probably means that the DB is not configured16:44
nowenare you sure this back up had users etc?16:44
Guest43741no16:44
Guest43741 this is the new server16:44
Guest43741this is the one with nothing but updates on it16:44
Guest43741we need to copy the data from the working server to this new one16:45
nowen I though the plan was to try to update the working server16:45
Guest43741no that did not work16:47
Guest43741tried twice but failed16:47
Guest43741with yum updates16:47
nowenso, to be clear - this 'working server' cannot be yum-updated without the kernel issue?16:48
Guest43741right16:49
nowenon the working server, please run 'rpm -qa | grep kernel'16:49
nowenis it the same? with all those kernels listed?16:55
Guest43741no16:59
Guest43741soem are same; soem are different16:59
Guest43741all kernels with 2.6.18-274 on old server are replaced on the new server with 2.6.18-308xxx17:00
Guest43741can we replicate between the two17:02
Guest43741lunch break... brb17:06
Guest43741ok18:03
Guest43741shall we continue?18:03
nowenyes. can you post the out of that rpm command to pastebin.com?18:04
Guest43741how do i do that in linux...18:04
Guest43741can i copy and paster18:04
nowenyes18:04
Guest43741so how do i copy?18:05
Guest43741can i send you screen shot18:06
nowenso, I'm guessing that the interface for hyper-v won't let you just cut and paste?18:08
nowenhow did you do it yesterday?18:08
Guest43741typed it in18:08
Guest43741can i email you the screen shot18:09
nowenis there a menu item on the console called clipboard?18:10
Guest43741yes but it does not work18:12
nowenthere are probably some guest drivers/additions to add18:14
nowenyou can also use putty to SSH into the servre18:15
nowenok - anyway - what is your plan agai?18:15
Guest43741that is what i'm doing right now18:18
nowenupgrading wikid via the rpms?18:20
Guest43741ok here is the link... http://pastebin.com/Mkv6EL0118:20
Guest43741so what do you think18:23
nowenlooks ok i guess18:24
Guest43741what we need to do now is copy the data from the working server to the new server18:25
Guest43741i think you tried via replication before but ran into some issues18:26
Guest43741should i try again or upgrade the new server first then replicate18:26
nowenI would say upgrade the new server first18:26
Guest43741ok lets try18:27
*** perkiset has quit (Ping timeout: 245 seconds)18:32
*** Guest43741 has quit (Ping timeout: 245 seconds)18:32
*** Mo (4084d7c2@gateway/web/freenode/ip.64.132.215.194) has joined #wikid18:32
*** Mo is now known as Guest9009718:32
Guest90097got disconnected for some reason18:33
Guest90097anyway18:33
Guest90097can you tell me which commands to run18:33
Guest90097or which version to install18:33
nowensure18:36
nowen'wget http://wikidsystems-dl.com/wikid-server-enterprise-3.5.0.b1330-1.noarch.rpm18:38
nowenand wikid-utilities-3.4.2-1.i386.rpm and  wikid-appliance-3.5.0.b1342-1.noarch.rpm at the same url18:38
nowenyou know, if you ssh into the server, you could cut and paste these commands18:38
nowenthen 'rpm -Uvh wikid-*' should upgrade them all18:39
Guest90097ok thanks running first one... going forward will ssh...18:40
Guest90097finished upgrading... however i still see ' The database is not yet configured for WiKID. Please run /opt/wikid/sbin/load_db.sh'18:43
Guest90097is this okay18:43
Guest90097and should i start wikidctl18:44
nowenhuh18:44
nowenI guess that brings up the question - are the users really in this server?18:44
Guest90097No this is the new server... freshly installed from ISO and updated will all patches... THAT IS IT...18:45
Guest90097not configured at all with any users18:45
nowenok18:45
nowenso, the plan is to copy over the db from the working server?18:46
Guest90097YES...18:46
Guest90097YES..18:46
nowenbut does the working server have the users?18:46
Guest90097is there another way???18:46
Guest90097yes... the working server is operational18:46
nowenok18:47
nowenyes, run wikidctl setup18:47
nowenand start the server18:47
Guest90097looks like it started18:51
nowencan you browse to the WiKIDAdmin?18:51
Guest90097yes... but i get 404 error18:52
nowenwhat does 'ls -all /opt/WiKID/tomcat/webapps/' list?18:52
Guest90097this is where we started having problems last time18:53
Guest90097list 5 files18:54
Guest90097root, wikid wikidadmin18:54
Guest90097wikidadmin.war and wikid.war18:54
nowenplease run 'rpm -qa | grep wikid'18:54
Guest90097ok ran18:57
Guest90097here is what i get18:57
Guest90097wikid-utilities-3.4.2-118:57
Guest90097wikid-server-enterprise-3.5.0.b1330-118:57
Guest90097wikid-appliance-3.4.21.b126-118:58
Guest90097wikid-appliance-3.5.0.b1342-118:58
Guest90097thats it18:58
nowenhm18:58
nowenrun 'ls -all /opt/WiKID/bin'18:58
Guest90097bunch of files... pastebin link: http://pastebin.com/mcynm4GM19:00
nowenhmm19:00
nowenrun 'wikidctl restart' and see19:01
noweneverything looks ok19:01
Guest90097same result:  HTTP status 40419:03
Guest90097description The requested resource () is not available.19:03
nowenrun '/opt/WiKID/bin/jsvc'19:03
nowendoes it return an error?19:05
Guest90097don't you have to run something in from of the above command19:06
nowenno19:06
nowennot if it starts with /19:06
nowenif you are in /opt/WiKID/bin, then you would do ./jsvc19:06
Guest90097ok i get 'No class specified'19:06
Guest90097'Cannot parse command line arguments'19:06
nowenok,run 'more /opt/WiKID/tomcat/logs/catalina.out' and see if there are any error messages19:07
Guest90097SEVERE: Exception opening database connection19:08
Guest90097PSQLException: FATAL: database "wikid" does not exist19:09
nowenhmm.19:09
Guest90097org.postgresql.util.PSQLException: Backend start-up failed: org.postgresql.util.19:09
nowenok19:09
nowenrun 'service postgresql start'19:10
nowendoes it start?19:10
Guest90097ok started19:10
nowenand you ran 'wikidctl setup' before starting, right?19:10
Guest90097yes19:11
nowenrun '/opt/wikid/sbin/load-db.sh'19:11
Guest90097of course19:11
Guest90097brb... getting coffee...19:12
Guest90097ok back19:39
Guest90097i get no such file for directory running /opt/... as per above19:40
nowenrun 'ls -all /opt/WiKID/sbin/19:40
Guest90097http://pastebin.com/rRpHhtZg19:43
Guest90097pasted to pastebin above19:43
nowenhmm, it's there19:44
nowen/opt/wikid/sbin/load-db.sh19:44
nowentry again19:44
nowenoh19:44
nowentry /opt/WiKID/sbin/load_db.sh19:44
Guest90097question... b4 upgrading should i not have db setup19:45
Guest90097?19:45
nowenthis will set up the db19:45
Guest90097that is, should i not have first started wikidctl so that it creates the db the do the upgrade?19:45
nowendidn't you run 'wikidctl start'?19:45
nowenand setup?19:45
nowenoh - I see what you are asking19:46
nowenno.  no need19:46
Guest90097ok19:46
Guest90097ran the above command19:47
Guest90097still no such file or d19:47
nowentry this: 'cd /opt/WIKID/sbin'19:48
nowenthen './load_db.sh'19:48
Guest90097you had load"dash" before now you have load"underscore"19:49
Guest90097ran but nothing seems to happen19:49
Guest90097should it have started a service19:49
nowenok19:49
nowenno19:49
nowenrun 'wikidctl start'19:49
Guest90097itwas already started19:50
Guest90097same http 40419:50
Guest90097should i restrt wikdctil19:52
nowenyes19:53
Guest90097restarted20:00
Guest90097same result from different machines and different browsers20:02
nowenok - let me do a few tests20:05
nowenhmm, going from a 1030 iso build and upgrading via rpm works fine for me20:11
nowenalthough you also ran yum update, right? was that before the rpm update?20:12
Guest90097yes20:13
Guest90097before the rpm update20:14
nowendoesn't seem like it would matter20:14
Guest90097installed server, ran yum updates20:14
nowenhmm  - run 'getenforce'20:14
Guest90097i'm sbin direcotory20:16
Guest90097how do i go back20:16
nowen' cd ..20:16
nowen'20:16
Guest90097ran 'getenforce'20:17
nowenpermissive?20:17
Guest90097all it says is 'permissive'20:17
Guest90097yes20:17
nowenok20:17
Guest90097file replciation20:33
Guest90097anything20:33
Guest90097ignore file replication20:34
Guest90097was trying to type into browser20:34
Guest90097thanks20:34
Guest90097anything?20:34
nowendoing a couple of yum updates20:35
nowenwell, that one works after yum update20:40
nowendid you reboot after the update?20:57
nowenGuest90097: works for me21:16
nowenGuest90097: I installed via the iso and ran these commands: http://pastebin.com/4FdisCQu21:17
nowenOK - time for me to vacate!22:35
*** nowen has quit (Quit: Leaving.)22:35

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!