| ionepoch | hmm.. | 00:00 |
|---|---|---|
| ionepoch | my paste got botched... | 00:00 |
| ionepoch | "/opt/WiKID/sbin/load_db.sh" | 00:00 |
| ionepoch | there we go.. | 00:00 |
| ionepoch | obviously proceed cautiously if your db is already setup... | 00:00 |
| ionepoch | you don't want to clobber anything | 00:00 |
| ionepoch | let me know if you can even get into your db using psql | 00:01 |
| dystie | ok. this is a production wikid server -- i would get yelled at a lot if I regenerated the cert. | 00:01 |
| dystie | 'm gonna see if I can get the stored password | 00:01 |
| ionepoch | yeah... don't do that. | 00:01 |
| ionepoch | nutshell.. | 00:02 |
| ionepoch | make sure you can connect to your db using psql | 00:02 |
| ionepoch | then make sure wikid can connect | 00:02 |
| ionepoch | if not... | 00:02 |
| ionepoch | you'll need to update the pass that wikid uses to connect | 00:02 |
| ionepoch | ... (i'm actually going to have to do that now... as i'm in the intial install I used a BS weak password for testing.. need to spruce it up) | 00:03 |
| dystie | lol. i keep meaning to set it up at home so i can be less pwn worthy. | 00:03 |
| dystie | funny how lazy i get even being a securitygeek. | 00:03 |
| *** perestre1ka has quit (Ping timeout: 252 seconds) | 00:04 | |
| ionepoch | let me know if you find the file that wikid uses in it's settings to connect to the db... | 00:07 |
| ionepoch | (i'm looking for it now) | 00:07 |
| *** perestrelka (~vladdy@194.242.5.47) has joined #wikid | 00:07 | |
| ionepoch | ok... | 00:09 |
| ionepoch | so | 00:09 |
| *** dystie has quit (Ping timeout: 245 seconds) | 00:10 | |
| *** dystie (c7ff532e@gateway/web/freenode/ip.199.255.83.46) has joined #wikid | 00:23 | |
| dystie | hi. ack had to reboot | 00:23 |
| ionepoch | ok... | 00:23 |
| ionepoch | so... trying to figure this out.. | 00:23 |
| ionepoch | it appears to me | 00:24 |
| ionepoch | that wikid during the setup phase needs postgres user's password to load the schema and add wikid users.. | 00:24 |
| ionepoch | the passwords for the wikiduser appear to be defaults... and set in | 00:24 |
| ionepoch | "/opt/WiKID/conf/WikidCode.properties" | 00:24 |
| ionepoch | wikid seems to have made the following users: | 00:25 |
| ionepoch | timecop | 00:25 |
| ionepoch | tomcat | 00:25 |
| ionepoch | wikid-sens | 00:25 |
| ionepoch | wikidadmin | 00:25 |
| ionepoch | wikidncserver | 00:25 |
| ionepoch | wikidradserver | 00:25 |
| ionepoch | in theory... wikid should have locked these down to local host... however... i need to double check this list | 00:26 |
| ionepoch | hostallall127.0.0.1/24trust | 00:26 |
| ionepoch | hostallall::1/128trust | 00:26 |
| ionepoch | hosttemplate1all::1/128trust | 00:26 |
| ionepoch | hosttemplate1all127.0.0.1/24trust | 00:27 |
| ionepoch | hostwikidall::1/128trust | 00:27 |
| ionepoch | hostwikidall127.0.0.1/24trust | 00:27 |
| ionepoch | localallalltrust | 00:27 |
| ionepoch | localpostgrespostgrestrust | 00:27 |
| ionepoch | localtemplate1alltrust | 00:27 |
| ionepoch | localwikidalltrust | 00:27 |
| ionepoch | cat /etc/postgresql/8.4/main/pg_hba.conf | 00:27 |
| ionepoch | so... i guess if you are having trouble connecting to the db | 00:27 |
| ionepoch | find your pg_hba.conf file on centos ... | 00:27 |
| ionepoch | and make sure it has enteries for wikid | 00:27 |
| ionepoch | if you have any error messages in ls /opt/WiKID/log/*.log ... or /opt/WiKID/tomcat/logs/catalina.out | 00:30 |
| ionepoch | paste them here | 00:30 |
| ionepoch | (make sure to strip out any passwords in case the are in the error message (but i doubt it)) | 00:30 |
| dystie | kk checking | 00:33 |
| dystie | i can't switch to the postgres user, there's no shell | 00:35 |
| dystie | (is dev/null | 00:35 |
| dystie | 4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later. log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later. log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later. log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later. log4j:ERROR Could not connect to remote log4j server at [localhost | 00:39 |
| dystie | was in dbmigrations.log | 00:39 |
| dystie | Exception in thread "main" com.sampullara.db.MigrationException: Some other failure to connect: jdbc:postgresql://localhost/wikid, {package=com.wikidsystems.db.migrations, passwd (not providing) | 00:40 |
| *** perestrelka has quit (Ping timeout: 248 seconds) | 00:40 | |
| dystie | Exception in thread "main" com.sampullara.db.MigrationException: Some other failure to connect: jdbc:postgresql://localhost/wikid, {package=com.wikidsystems.db.migrations, user=postgres, password=REMOVED, url=jdbc:p | 00:40 |
| dystie | is full string | 00:40 |
| *** perestrelka (~vladdy@194.242.5.47) has joined #wikid | 00:41 | |
| ionepoch | hmmm.. | 00:48 |
| ionepoch | the log 4j error doesn't matter | 00:48 |
| ionepoch | that's just for loggin... | 00:48 |
| ionepoch | but.. | 00:48 |
| ionepoch | the last error looks bad | 00:48 |
| ionepoch | ... | 00:48 |
| ionepoch | can you find the pg_hba.conf on your server? | 00:48 |
| ionepoch | you should have a line like: local postgres postgres trust | 00:49 |
| ionepoch | try this too: psql -Upostgres | 00:50 |
| dystie | a bunch of java errors follow, let me know if you want stuff posted | 00:50 |
| ionepoch | that way you don't need a postgres | 00:51 |
| ionepoch | try this first | 00:51 |
| ionepoch | psql -Upostgres | 00:52 |
| dystie | found the file | 00:52 |
| dystie | what user should it be run as? | 00:52 |
| dystie | it was whining when i tried to run as root. | 00:52 |
| ionepoch | shouldn't matter | 00:52 |
| ionepoch | huh... | 00:52 |
| ionepoch | k | 00:52 |
| ionepoch | well | 00:52 |
| ionepoch | i ran it as root | 00:52 |
| ionepoch | try psql -Upostgres template1 | 00:52 |
| ionepoch | ie launch psql as user:postgres connect to db template1 | 00:52 |
| dystie | onesec. | 00:53 |
| ionepoch | that's what psql -Upostgres template1 does | 00:53 |
| dystie | # psql -Upostgres template1 psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? | 00:53 |
| ionepoch | ouch | 00:53 |
| ionepoch | so | 00:53 |
| ionepoch | yeah | 00:53 |
| ionepoch | that would be your first problem.. | 00:54 |
| ionepoch | type .. ps ax | grep post | 00:54 |
| ionepoch | i get .. 13825 ? S 0:00 /usr/lib/postgresql/8.4/bin/postgres -D ..... more follows | 00:54 |
| ionepoch | basically ps ax just looks for running processes... | 00:54 |
| ionepoch | looks like your database isn't on | 00:54 |
| dystie | root 4100 0.0 4.9 378044 25280 pts/0 Sl 19:38 0:00 /opt/java/bin/java -cp /opt/WiKID/lib/xstream-1.2.jar:/opt/WiKID/lib/xpp3_min-1.1.3.4.O.jar:/opt/WiKID/lib/xmlsec-1.4.0.jar:/opt/WiKID/lib/xml-apis.jar:/opt/WiKID/lib/xercesImpl.jar:/opt/WiKID/lib/wsdl4j-1.5.1.jar:/opt/WiKID/lib/wikid-server-enterprise-lib-3.4.87.jar:/opt/WiKID/lib/smtp.jar:/opt/WiKID/lib/shared-ldap-0.9.5.4.jar:/opt/WiKID/lib/saaj.jar:/opt/WiKID/lib/rads | 00:55 |
| ionepoch | hmmm | 00:55 |
| ionepoch | let's make that more specific... | 00:55 |
| ionepoch | ps ax | grep postgres | 00:55 |
| ionepoch | basically... just looking to see if you db is running... doesn't look like it.. | 00:56 |
| ionepoch | how about: | 00:56 |
| ionepoch | root@wikid:~# /etc/init.d/postgresql-8.4 status | 00:56 |
| ionepoch | Running clusters: 8.4/main | 00:56 |
| dystie | [root@noneya) data]# ps aux | grep postgres root 4100 0.0 4.9 378044 25280 pts/0 Sl 19:38 0:00 /opt/java/bin/java -cp /opt/WiKID/lib/xstream-1.2.jar:/opt/WiKID/lib/xpp3_min-1.1.3.4.O.jar:/opt/WiKID/lib/xmlsec-1.4.0.jar:/opt/WiKID/lib/xml-apis.jar:/opt/WiKID/lib/xercesImpl.jar:/opt/WiKID/lib/wsdl4j-1.5.1.jar:/opt/WiKID/lib/wikid-server-enterprise-lib-3.4.87.jar:/opt/WiKID/lib/smtp.jar:/opt/WiKID/lib/shared-ldap-0.9.5.4.jar | 00:56 |
| dystie | running that now | 00:56 |
| dystie | postmaster is stopped | 00:57 |
| ionepoch | yeah... | 00:57 |
| ionepoch | k | 00:57 |
| ionepoch | so that's the biggy | 00:57 |
| ionepoch | ... | 00:57 |
| dystie | starting service failed | 00:57 |
| ionepoch | k | 00:57 |
| ionepoch | so looks like a problem with the db | 00:57 |
| ionepoch | do you know if any centos updates were applied... | 00:58 |
| dystie | yeah, imagine that we just patched the box | 00:58 |
| dystie | we patched our other wikidboxes w/ (i think) the same patches.. and they're happy. | 00:58 |
| ionepoch | shouldn't happen... but i guess it's possible that a patch could have caused the db tables to be upgraded... and perhaps the updates failed? | 00:58 |
| ionepoch | that's a little presumptuous on my part | 00:58 |
| dystie | nah, is not presumptious. i'm checking to see what patches we pushed. | 00:59 |
| ionepoch | tail -1000 /var/log/postgresql/postgresql-8.4-main.log | 00:59 |
| ionepoch | find your postgres log | 01:00 |
| ionepoch | see if there are any errors in there | 01:00 |
| dystie | pgstartup.log? | 01:04 |
| dystie | FATAL: lock file "postmaster.pid" already exists HINT: Is another postmaster (PID 5323) running in data directory "/var/lib/pgsql/data"? LOG: logger shutting down LOG: logger shutting down LOG: logger shutting down LOG: logger shutting down LOG: logger shutting down | 01:05 |
| dystie | looking for this pid file | 01:05 |
| dystie | no postgresql.pid in teh data directory | 01:08 |
| dystie | in the directory it was whining about. | 01:08 |
| ionepoch | look in /var/run | 01:08 |
| ionepoch | or | 01:08 |
| ionepoch | as a longshot | 01:09 |
| ionepoch | ... /etc/init/postgres zap | 01:09 |
| ionepoch | (zap is supposed to delete pids) | 01:09 |
| ionepoch | (not your db) | 01:09 |
| ionepoch | Usage: /etc/init.d/postgresql-8.4 {start|stop|restart|reload|force-reload|status|autovac-start|autovac-stop|autovac-restart} | 01:10 |
| ionepoch | maybe force-reload | 01:10 |
| dystie | force reload ran w/no errors | 01:14 |
| ionepoch | hmm | 01:14 |
| ionepoch | can you start it now? | 01:14 |
| dystie | no, did not start | 01:14 |
| ionepoch | what's in the db error log? | 01:15 |
| ionepoch | same error? | 01:15 |
| ionepoch | basically we need to kill that pid file and get your db started | 01:15 |
| dystie | yeah and it's not where i'd think it'd be | 01:16 |
| dystie | just cleared log to see current stuff. waiting | 01:16 |
| dystie | runuser /dev/null -- permission denied is all that's logged | 01:16 |
| ionepoch | hmm | 01:17 |
| ionepoch | are you out of diskspace? | 01:17 |
| ionepoch | df -h | 01:17 |
| ionepoch | and are you starting the db as root? | 01:17 |
| dystie | nothing is running at 100% | 01:18 |
| dystie | yeah i'm in a root shell | 01:18 |
| ionepoch | so | 01:18 |
| dystie | heh. looks like postgres doesn't have a shell on the server that's cranky. and does on the server in another env that isn't cranky. | 01:19 |
| ionepoch | back to what you mentioned earlier... | 01:19 |
| ionepoch | yes | 01:19 |
| ionepoch | that's what i'm thinking | 01:19 |
| ionepoch | change shell to bin/sh | 01:19 |
| ionepoch | or something | 01:19 |
| dystie | checking patching ticket to make sure i don't get yelled at. onesec. btw, you totally rock; if you're ever @ like. shmoo or defcon i'll buy you a beer. | 01:19 |
| ionepoch | ha! | 01:20 |
| ionepoch | yes | 01:20 |
| ionepoch | hang with shellfish at defcon in last years competition | 01:20 |
| ionepoch | http://www.defcon.org/html/defcon-19/dc-19-contest-results.html | 01:22 |
| dystie | [root@chi-esx-v35:(chi-otp-01) pgsql]# rpm -qa | grep 'postgres' postgresql-libs-8.1.23-1.el5_6.1 postgresql-libs-8.1.23-1.el5_7.3 postgresql-jdbc-8.1.407-1jpp.4 postgresql-libs-8.1.23-1.el5_7.3 postgresql-8.1.23-1.el5_7.3 postgresql-8.1.23-1.el5_6.1 postgresql-server-8.1.23-1.el5_6.1 postgresql-pl-8.1.23-1.el5_6.1 postgresql-server-8.1.23-1.el5_7.3 postgresql-pl-8.1.23-1.el5_7.3 | 01:24 |
| dystie | ack. dumb. | 01:24 |
| dystie | so that's what we got -- but on a working server in the other domain i have: | 01:24 |
| dystie | postgresql-jdbc-8.1.407-1jpp.4 postgresql-libs-8.1.23-1.el5_7.3 postgresql-libs-8.1.23-1.el5_7.3 postgresql-8.1.23-1.el5_7.3 postgresql-server-8.1.23-1.el5_7.3 postgresql-pl-8.1.23-1.el5_7.3 | 01:24 |
| dystie | schweet. yeah, i spent most of my time in skytalks. was quieter. | 01:25 |
| ionepoch | hmm... is your db booting now? | 01:25 |
| ionepoch | can't tell from the patch string if your server is having issues or not | 01:25 |
| ionepoch | ... | 01:25 |
| ionepoch | just looks like your 02 box has been around a bit longer? | 01:26 |
| ionepoch | migrated from 5.61 to 5.73 | 01:26 |
| dystie | working or not working? | 01:26 |
| dystie | i don't know which one got spun up first. yeah, stuff was upgraded. | 01:27 |
| dystie | looks like on centos postgresql likes to live in /var/lib/pgsql | 01:27 |
| ionepoch | ahh | 01:27 |
| ionepoch | ... | 01:27 |
| ionepoch | it's booting though right? | 01:27 |
| dystie | and theoretically the lock file should be in /var/lib/pgsql/data but i'm not finding it | 01:27 |
| ionepoch | all is well? | 01:27 |
| dystie | not so much | 01:29 |
| dystie | i'm changing the shell. and hoping i don't get yelled at. | 01:29 |
| ionepoch | well... for the short term... i'd change it just to see if you can get the db to go... (pretty sure the init script needs it)... | 01:30 |
| dystie | no you're spot on | 01:30 |
| ionepoch | if it works ... you can turn the shell off... and regroup with your team to discuss | 01:30 |
| dystie | changing shell | 01:30 |
| dystie | Starting postgresql service: [ OK ] Starting postgresql service: [ OK ] Starting postgresql service: [ OK ] | 01:34 |
| dystie | happy. k. | 01:34 |
| ionepoch | heh | 01:34 |
| ionepoch | cool | 01:34 |
| ionepoch | now to the rest... how's about your wikid | 01:34 |
| dystie | looks like it started -- reloading page | 01:35 |
| dystie | cool, fixing failover server. | 01:35 |
| dystie | i owe you like. two beers. | 01:35 |
| ionepoch | we love beer :) | 01:36 |
| dystie | yus. tasty tasty malty goodness | 01:36 |
| ionepoch | well, we plan to be at defcon competing again this year... | 01:36 |
| dystie | i plan on being there volunteering for skytalks (well at least some of the time.) | 01:36 |
| ionepoch | so if you make it... stop by the competition and look for shellphish | 01:37 |
| ionepoch | you going to blackhat? | 01:37 |
| dystie | nah, i want to blow my training budget on sans this year | 01:37 |
| dystie | i got promoted past my competence level, so i need to catch up | 01:37 |
| ionepoch | cool! always room to rise to the occasion | 01:38 |
| ionepoch | cheers... gonna try and wrap this up so i can head out... probably be in here tomorrow for a bit as well | 01:38 |
| dystie | coolbeans | 01:40 |
| dystie | yeah, i'm happy stuff is working. gonna go get some f00d. | 01:40 |
| dystie | where are you in the US? | 01:40 |
| ionepoch | Cali / westside | 01:40 |
| dystie | coolios. 'm eastcost/virginia. | 01:41 |
| dystie | not as cool as cali. | 01:41 |
| ionepoch | sorry to do this... but today was freaking amazing weather wise (did i just lose a beer?) | 01:41 |
| *** dystie has quit (Ping timeout: 245 seconds) | 01:45 | |
| *** mick_lap1op (~mick@mickweiss.com) has joined #wikid | 06:50 | |
| *** mick_laptop has quit (Read error: Connection reset by peer) | 06:50 | |
| *** ionepoch has quit (Ping timeout: 340 seconds) | 07:15 | |
| *** mick_lap1op has quit (*.net *.split) | 08:29 | |
| *** mick_laptop (~mick@mickweiss.com) has joined #wikid | 08:40 | |
| *** mick_laptop has quit (Remote host closed the connection) | 09:37 | |
| *** mick_laptop (~mick@mickweiss.com) has joined #wikid | 09:37 | |
| *** mick_laptop has quit (Read error: Connection reset by peer) | 10:06 | |
| *** mick_lap1op (~mick@mickweiss.com) has joined #wikid | 10:07 | |
| *** FlexyZ (3e74c0c6@gateway/web/freenode/ip.62.116.192.198) has joined #wikid | 13:24 | |
| FlexyZ | hey | 13:24 |
| *** FlexyZ has quit (Ping timeout: 245 seconds) | 14:15 | |
| *** FlexyZ (3e74c0c6@gateway/web/freenode/ip.62.116.192.198) has joined #wikid | 14:33 | |
| FlexyZ | hi | 14:36 |
| *** FlexyZ has quit (Ping timeout: 245 seconds) | 15:32 | |
| *** ionepoch (~ionepoch@wsip-98-173-30-75.sb.sd.cox.net) has joined #wikid | 17:18 | |
| ionepoch | bump... wondering if there is a way to disable the automatic redirect on port 80 to the wikidadmin | 17:18 |
| ionepoch | also wondering if anybody can try and preload themselves as a user to port 80.... (in other words... can i disable the ability for people to try and self pre-register?) | 17:20 |
| ionepoch | bump | 18:26 |
| ionepoch | anyone around | 18:26 |
| ionepoch | ? | 18:26 |
| ionepoch | having a bit of difficulty getting radius server included in the enterprise edition to return a group attirbute | 18:27 |
| ionepoch | any pointers? | 18:27 |
Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!