Thursday, 2013-02-07

*** nowen (~nowen@50-194-249-125-static.hfc.comcastbusiness.net) has joined #wikid13:51
nowenjlkinsel: any progress?15:04
*** dystia (c7ff5332@gateway/web/freenode/ip.199.255.83.50) has joined #wikid15:45
dystiayo nick.15:46
nowenhey15:46
dystiai need some guidance;15:46
nowenok15:46
dystiathis15:46
dystiaack15:46
dystiaok.   /var/lib/pgsql/data/base/16817 has grown to 1.6gb15:47
dystiaand it's filling the file system15:47
dystiawhat's the purpose of the file and can it be purged and how.15:47
nowenis your log page getting slow?15:47
dystiayes actually.  i wasn't able to pull logs last night.15:48
nowenok - go to that page, if you can and archive the logs15:48
dystia84246915:48
dystiamisfire.15:48
dystiaok checking15:48
dystiaslow like whoa.15:49
dystia Download Creation Date Size (bytes) Delete 1360252243595.zip2013-02-07 10:500[X] 1360252267823.zip2013-02-07 10:510[X]15:51
dystiashowing as 0 when i click thearchive button - does that update as it runs the archive job/15:51
nowenthe size is 0?15:52
dystiayeah, it shows up in the Current archives on server: as 015:53
dystiabut the dir contains 1.5gb of files15:53
nowenit should list a size15:53
dystiaof 0 bytes, yes15:53
dystiawhere do these files get placed on the fs?15:54
nowen/opt/WiKID/tomcat/webapps/WiKIDAdmin/logArchive/15:54
dystiai tried to download it and it said it was 0 bytes in the download window15:54
dystiathey're showing as 0 on the fs.15:54
nowenmaybe there's not enough disk space for it?15:55
dystiawell i'd expect it to grow and then fail.15:55
dystiathe files are staying at 015:55
dystia 340M is free on /opt15:55
nowentry older than 2 months or something15:56
dystiai tried older than 2 hours orig15:57
dystiaand now i'm down to 4% free on the problem fs.15:57
nowencan you run 'yum clean all'15:57
nowenmight free up some space15:57
dystiaonesec.  gotta file a emergency change15:58
dystiaok.  looks like i have a few minutes on this one;  it's not as full as it was for some reason.15:59
dystiahoping i didn't click delete logs w/out archiving.  :)15:59
dystiaand nope, it's full.  :)16:00
dystiak.  filing emergency change.16:00
nowenanything in /root/ ?  old rpms that can be deleted?16:01
dystiait's not the same fs.16:01
nowenok16:02
dystia var is on its' own partition16:02
dystiadoes yum keep it's files in /var16:03
dystiabecause 1.5gb of the files on var are in that one dir.16:03
nowennot sure, but I think so16:03
dystiayeah.  /var/cache/yum is default16:04
dystiait freed up like 60mb16:27
dystiawhen i ran yum clean all.16:27
dystiawhat should I do next?16:27
nowenI think try to archive some lesser amount, like older than 4 months16:28
dystiaanother 0 byte file w/ that archive setting16:30
nowenis there an error message in the logs?16:30
dystiawikid-server-enterprise-3.5.0-b1342  is what version the server is on, btw16:33
nowenthat's pretty up-to-date.16:33
nowenare the loggers set to debug?16:34
dystiaset the query to debug16:34
dystianowen@wikidsystems.com?16:34
dystiai'd rather email the log16:35
nowenis there an error?16:35
dystiathere's stuff i don't understand.16:35
dystiabut none is red16:35
nowenthat's the email16:35
dystiaHTTP Access Logger is warn16:36
dystiacom.wikidsystems is info16:36
dystiacom.wikidsystems.client.wClient  is warn16:36
dystiacom.wikidsystems.server.wAuth is info16:36
dystiaorg.apache is warn16:36
nowenok, we can try vacumming the db16:37
dystiasteps?16:38
nowen# su - postgres16:39
nowen$ psql -d wikid16:39
nowenwikid=# SELECT pg_database_size('wikid');16:39
nowenthat will show the size16:40
nowenwikid=#  VACUUM FULL;16:40
nowenthat will vacuum it16:40
nowenit should return VACUUM16:40
nowenthen run the size command again16:40
dystiapsql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"?16:41
nowenoh, are you running in replication?16:41
dystiayes16:41
dystiareplicates to an -02 box16:41
nowenadd -p 543416:42
dystiato?16:42
dystiasyring16:42
nowenthe psql -d wikid -p 543216:42
dystiapsql -p 5434 -d wikid16:42
dystiakk16:42
dystiapsql -d wikid -p 543216:42
dystiapsql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"?16:43
nowenthe server is up, right?16:43
nowenctrl-d back to root and run 'netstat -anp | grep 543' and see what port it is on16:43
dystiacheck your email16:44
nowenok - try using -p 543416:45
nowenmy bad16:45
dystiaERROR:  syntax error at or near "wikid" at character 1 LINE 1: wikid=# SELECT pg_database_size('wikid');16:45
dystia postgres  | postgres | UTF8  template0 | postgres | UTF8  template1 | postgres | UTF8  wikid     | postgres | SQL_ASCII16:47
dystiaare the dbs in the db16:47
nowentry the same command again16:47
dystiasame error16:47
nowenhmm16:47
nowenthe db is there16:48
nowenit is exactly thje command I'm using16:48
nowenwikid=# SELECT pg_database_size('wikid');16:48
nowen pg_database_size16:48
nowen------------------16:48
nowen          562855216:48
nowen(1 row)16:48
dystiado you need to do a \connect wikid first16:49
nowenwair16:49
nowenwait16:49
nowenjust type 'SELECT pg_database_size('wikid');'16:49
nowen;-)16:49
dystiawikid=# 'SELECT pg_database_size('wikid');' wikid-#16:50
nowenif you do 'psql -d wikid -p 5434' then you are in wikid16:50
dystiaah.  doh16:51
nowenlol16:51
dystiathe query returned blank16:51
dystiai don't do dbs dude.  :P16:51
nowenI hear ya16:52
dystiai hate them.  i make the dbas do that.16:52
nowenthis is the extent of mine16:52
dystiadatabase security is always epic fail.16:52
dystiaso i did a \d and got a list of tables16:52
dystiathe pg_database_size was not a listed type/name/in the shcema16:52
dystiaalso when i run the query it returns blank16:52
nowendid the command not work?16:54
dystiawikid-# 'SELECT pg_database_size('wikid');' wikid-#16:54
dystiaall it does is return to the next line16:54
nowenSELECT pg_database_size('wikid');16:54
dystiait does not tell me anything.16:54
nowenis all you need16:54
dystiaERROR:  syntax error at or near "'SELECT pg_database_size('" at character 1 LINE 1: 'SELECT pg_database_size('wikid');'         ^ wikid=# SELECT pg_database_size('wikid');  pg_database_size  ------------------        1540738940 (1 row)16:54
dystianow we're talking.16:54
dystiaand that size looks right for my problem.16:55
nowenok16:55
dystiavaccuming16:55
nowenso now run VACUUM FULL;16:55
dystiak.  is slow.16:56
dystia wikid=# VACUUM FULL; VACUUM wikid=# SELECT pg_database_size('wikid');  pg_database_size  ------------------        1534251012 (1 row)16:56
dystiais that nubmer supposed to be higher?16:57
dystiavar has not gone down in utilization16:58
nowenhmm16:58
dystiamy boss wants a root cause analysis when we're done, btw.16:59
nowenmine went down from 7578248 to 562855216:59
dystiai want to make sure the service is stable first obviously.16:59
dystiarerun vaccum?16:59
nowenyou're sending radius account packets to the WiKID server and it's choking on them16:59
nowenyes17:00
dystia       153425101217:02
dystiawhat next?17:02
nowenhow much space is on the fs?17:04
dystiaafter first vaccum17:04
dystia                      2.1G  1.9G  131M  94% /var17:04
dystiaafter second vaccum17:04
dystia                      2.1G  1.9G  113M  95% /var17:04
nowentry17:06
nowenREINDEX DATABASE wikid17:06
nowen;17:06
dystiawikid=# SELECT pg_database_size('wikid');  pg_database_size  ------------------        153431641217:07
dystiai ran REINDEX DATABASE wikid17:07
dystiait didn't want a ; n the end of the line17:07
dystiareturned a #wikid prompt17:07
dystiadid it want the ; on the next line?17:07
nowenyou can do it all on one line17:08
nowenor two17:08
dystiakk17:08
dystiaits' running17:08
dystiayay learning stuff.17:08
nowensorry you have to17:09
dystiame too.  :)17:10
dystiaERROR:  could not write block 1356 of relation 1663/16817/17506: No space left on device17:10
nowendamn17:10
dystiawhere next?17:10
nowengot any big log files in /var/log?17:11
dystiarw-------  1 root root  94M Jan 27 04:02 messages.2 -rw-------  1 root root  50M Jan 20 04:02 messages.3 -rw-------  1 root root  45M Feb  3 04:02 messages.1 -rw-------  1 root root  40M Feb  7 12:11 messages -rw-------  1 root root 2.1M Dec 23 04:02 messages.4.gz -rw-------  1 root root 1.8M Dec 30 04:02 messages.3.gz -rw-------  1 root root 1.4M Jan  6 04:02 messages.2.gz -rw-------  1 root root 1.3M Jan 13 04:02 messages.1.gz17:12
dystianot terribly17:12
dystiai have a snap - i can clean a few of the messages files without feeling too guilty17:12
dystia2.1G  1.7G  264M  87% /var17:13
dystiarerun the reindex?17:13
nowenyeah17:14
dystiaREINDEX wikid=# SELECT pg_database_size('wikid');  pg_database_size  ------------------        1483345924 (1 row)17:15
nowenok17:15
dystia                      2.1G  1.8G  191M  91% /var17:15
dystiawhere next?17:15
nowentry vacuuming again17:15
dystiakk17:15
dystiai have to do a vuln review at 1 that i cannot not go to17:15
dystiawhich means i may be handing you to someoen else, we'll see17:15
dystia pg_database_size  ------------------        148327219617:17
dystiano change17:17
dystiawe are now at17:18
dystia              2.1G  1.8G  240M  89% /var17:18
dystiaso a little bit more space.17:18
nowengotta an idea17:19
nowentesting it now17:19
dystiais this going to take long enough for me to justifiably go to the restroom and smoke a cig?17:19
nowensure17:20
nowenI don't want you jonesing!17:20
dystiakk brb17:20
nowendystia: you know, the log zips are going to /opt/ not /var17:27
dystiasure but they were showing as 017:31
dystiaso that wasn't what's killing /var17:31
nowenhmm, yeah, but if they get written, then the are removed from the db in /var17:31
nowencan you try archiving again?17:32
dystia1360258311490.zip2013-02-07 12:310[X]17:32
dystiasame issue17:32
dystiai can run a delete instead of a archive.17:32
nowenreally?17:32
dystiahad tried that w/ no results already.17:32
nowentry again17:33
* nowen crosses finger17:33
dystia                     2.1G  2.0G   54M  98% /var17:33
dystiavar is spiking again.17:33
nowenok17:36
nowenlet's delete some logs from the db directly17:36
dystiasyntax17:37
dystiacus now var is 100%17:37
dystiawikid=# SELECT pg_database_size('wikid');  pg_database_size  ------------------        1483337596 (1 row)17:37
dystiaand i can't get at the log page in the console.17:38
nowen delete from logging_event where logger_name = 'com.wikidsystems.radius.log.DBSvrLogImpl';17:38
dystiak.  running17:38
dystiayou know this makes me nervous.17:38
dystiabut i have a snap.17:38
nowenyeah. good idea17:38
dystiawhat are the implications for the replication server?17:39
nowenI think the best idea is to re-sync after this17:39
dystiawikid=# delete from logging_event where logger_name = 'com.wikidsystems.radius.log.DBSvrLogImpl'; DELETE 64570117:39
dystiawikid=# SELECT pg_database_size('wikid');  pg_database_size  ------------------        148333759617:39
nowenvacuum again plz17:39
dystia                      2.1G  1.8G  253M  88% /var17:39
dystiayou were saying the accounting messages are whats' killing it?17:40
nowenthat's what I think based on the log you sent me17:40
nowenyou might also have radius set to debug17:41
dystiawhere would i find that?17:41
dystiai'm running AD to openvpn and wikid17:41
nowenconfiguration / enable protocols / radius17:41
nowenif you want to delete all the events in the logs:17:42
nowendelete from logging_event;17:42
dystiaUse Accounting?: is checked17:43
nowenuncheck that17:43
dystiaDebug Level: Normal High Debug  is normal17:43
dystiawhat does unchecking that do?17:43
nowenyou want Normal17:44
dystiapg_database_size  ------------------         44721343617:44
nowenhow is the fs?17:44
dystia                     2.1G  758M  1.3G  38% /var17:44
dystiamuch better.17:44
nowenok, try zipping the rest via the WiKIDAdmin17:45
nowenhopefully it is also fater17:45
nowenfaster17:45
dystiawhat's the command to restart wikid?17:46
dystiasudo service wikid restart?17:46
nowenwikidctl restart17:46
dystiak17:46
dystiarestarting to apply the accounting change17:46
nowenalso, check to see of openvpn or anything is sending WiKID account packets17:47
dystia                2.1G  777M  1.2G  39% /var17:55
dystialooks like it's being stable for a minute, or at least not likely to fill up again in the next half hour17:55
dystiai need to run into this meeting;  i'll check in w/ you once i'm out and make sure openvpn isn't killing it.17:55
nowenno, it most likely took awhile to get this way17:55
dystiait  did.17:56
dystiakk. running tomeeting18:00
nowenlater18:00
*** dystia has quit (Ping timeout: 245 seconds)18:04
*** dystia (c7ff5332@gateway/web/freenode/ip.199.255.83.50) has joined #wikid20:15
dystiayo20:16
dystiaaight.  out of meeting.20:16
nowenhey20:16
dystiaso - impact of this.20:16
dystiai'm checking the host to see how disk is doing.20:16
nowenok20:17
dystiawtf caused this?  like - is this going to happen again, and is this a risk to my other systems.20:17
dystiathat was my main production env;  the only other way to get in is a trip to the datacenter.20:17
nowenI'm guessing that either the account on WiKID or accounting being sent to WiKID caused it20:17
dystiais the accounting normally an issue?20:18
dystia'account on wikid'?20:18
nowenmost people don't uses20:18
nowenis it20:18
nowenradius accounting, I mena20:18
dystiawhat's the purpose of radius accounting/20:19
nowennot sure, extra logging, I guess? You would only really use it on a real radius server. I will probably remove it as an option20:20
dystiawhat did you mean by 'account on wikid'?20:21
nowenI meant accounting on wikid, which you turned oof20:21
nowenlook at that log you sent me, it is almost all accounting20:22
dystiasure.20:22
dystiai'm pulling logs from the other two consoles.20:22
nowenwhat do you mean?20:23
dystiawell. 1)20:23
dystiaI need to be sure that if this is an issue in 1 environment, that it's really the issue, and that if so, we turn off accounting on the other two consoles to avoid replicating the issue.20:24
dystiabecause i don't want my OTHER prod env to have issues.20:24
dystia2 is I need to understand how this affects us in a replication setup.20:24
nowenahh - ok - I forget you have two20:24
dystiaso i'm checking disk space and logs on the other two consoles.20:24
dystiai have 3.  hence needign an IAM.20:24
dystia(no, a web portal is not going to work.)20:24
nowenyou can look at Gluu, josso and CAS. we've integrated login with all of those.  but not all the stuff you want.20:25
dystiawhats' been integrated?20:25
nowenjust login20:25
dystianot configuring someone/20:25
dystiawhats login?20:25
nowengluu may have more as they did the scripting via the api.  cas was radius, josso was login via wauth20:26
nowenentering a username and otp for validation20:26
nowenauthenticating20:26
dystiaso just logging into its console/20:26
dystianot provisioning accounts.20:26
nowenlogin to whatever. not provisioning20:27
dystiacheck.20:27
dystiajesus nick.20:27
dystiathis is killing me.20:27
nowenis it just because you have so much turn-over by consultants?20:28
dystiawell, it's because the request process takes a while to complete.20:28
dystialike.  AD account, THEN openvpn, THEN wikid20:28
dystiawikid involves generating a xlsx and puttign in fields in the right order (have horked before) and uploading as tab delinmited20:28
nowendo they need an account in openvpn?  if you're using radius?20:28
dystiaif someone botches it I have to save out hte info, clear the table, reinter it.20:28
dystiaif someone fucks up their token I created w/ prereg and have a completed reg in the table, i have to save out the info, clear out the table, reupload the data.20:29
dystiaif someone clicks clear the table w/out doig that it invalides all the prereg and its' not retreivable20:29
dystiaand that's how Ian has it now - we use openvpn for authoirzation/vpn profiles.20:29
dystianot everyone gets the same profile.20:29
dystiaconsultant turnover is not my issue.  my issue is that my company does not have the concept of 'hire a cheap bitch for that' and expects me to cover a helpdesk role, and it's complicated enough that it's hard to hand to an idiot.20:30
dystiathere are systems that can crank people out by business role.  i want push button based on ruleset and AD.20:30
nowenyeah, sounds like a confluence of a number of things.  no one else even remotely has these issues20:31
dystiai need IAM regardless of wikid but having to do the wikid stuff is definitely a part of the process i dislike.20:31
dystiasure - so how do other people make it not suck20:31
nowenmost use AD and WiKID. routing auth through AD and letting users reg themselves based on their AD creds20:32
nowenone prospect has written his own IAM and has his helpdesk adding users20:32
dystiai can't do firewall ruleset with AD - and not everyone should be able to go everywhere.20:33
nowenAD does authorization based on groups.20:33
nowenmaybe it's openvpn. you might be too bog for it20:34
dystiaabsolutely.   but it doesn't assign firewall rules.20:34
nowenwhere bog == big20:34
nowennot sure what happened to my fingers today20:34
dystiai can do that via groups/authorizations but i'd need a better vpn solution/20:34
dystiayeah, ian has issues.  <bitch>20:34
dystiathe probelm is the amount of time it takes to rip stuff out.20:34
dystia2013-02-07 12:48:43.514ERRORcom.wikidsystems.client.wClientERROR: java.net.SocketException: Broken pipe  2013-02-07 12:48:43.410ERRORcom.wikidsystems.server.wAuthCouldn't validate the client certificate. Verify the validity and dates of the client cert.20:36
dystiathat's new.20:36
nowenrun keytool on your certs: http://www.wikidsystems.com/support/wikid-support-center/troubleshooting-faq/how-do-i-know-if-my-certificate-is-valid20:36
dystiak.  if i leave it like htat for now is it likely to die/20:40
nowenyou can just create a new localhost cert20:41
dystiacus it neesd fixing but it looks like i need to turn off accounting for the other two consoles and another machine is having the same issue20:41
dystiayeah - i just hesitate to touch cacerts on a production server without talking to someone who plays w/ it more.20:41
dystia(not something i have a lot of experience w.)20:41
nowenare you able to login to openvpn using that server?20:42
dystiayeah auth is up.20:42
dystiais it likely to be a fire?20:42
nowenit would affect any service using the localhost cert such as example.jsp20:43
dystiak.20:55
dystiagr.  :)20:55
dystiaat least that's a fairly easy fix.  i'll see if i can get it tasked.20:55
dystiatell your dude who wrote his own IAM that I'm adorable and buy beer and want his code.20:56
nowenlol20:56
dystiai use the adorable bit a lot.20:56
dystiafrequently works.20:56
dystiaoh.  where on openvpn would i be seeing the accounting issue, or does it just pick up once it comes into wikid?20:56
dystia(is this something that cascades)20:56
nowenlook under the radius authentication piece20:57
dystiai turned off accounting for the server that was having issues.21:11
dystiaoh in vpn.checking21:11
*** nowen has quit (Remote host closed the connection)21:15
*** nowen (~nowen@50-194-249-125-static.hfc.comcastbusiness.net) has joined #wikid21:16
nowenhmm,  random X reboots21:33
*** Excid3LogBot (~Excid3Log@ec2-174-129-6-100.compute-1.amazonaws.com) has joined #wikid21:35
nowentesting another bot21:35
*** Excid3LogBot has quit (Read error: Connection reset by peer)21:37
*** Excid3LogBot (~Excid3Log@ec2-174-129-6-100.compute-1.amazonaws.com) has joined #wikid21:39
nowenhttp://foo.com21:39
dystiaso i need to talk to you about how wikid handles accounting21:40
dystiahow does wikid handle logging accounting data and reporting it back to the radius server?21:40
dystiabecuase my understanding is tha tkilling it loses me my logout tracking.21:41
nowenwell, WiKID has no concept of when someone logs out21:41
dystiaso why does it generate radius accounting data and where does it store it and where does it send it?21:42
nowenI don't think it stores it anywhere.  that's why I want to remove it21:44
dystiathen why is the DB filling up?21:47
dystiait shouldn't, iiuc appropriate radius behavior.21:47
dystiawhat is killing the log files?21:47
nowenwell, I was wondering if the openvpn server is sending it radius accounting packets21:48
nowencan your run tcpdump on port 1813?21:48
dystiaheh21:51
dystiatheoretically.21:51
dystiaonesec.  i'm looking at the radius configs on the domain controllers.21:51
dystiaif i'm not doing accounting logging there then its stupid to log it elsewhere.21:52
nowenthe logs you sent say "Accounting packet receive from xxx on NAS 10.20.x.x."21:52
nowenI was assuming that the NAS was openvpn21:52
dystiaAccounting packet receive from bleh on NAS21:54
dystiaso what is wikid doing when it logs that?21:54
nowenjust logging it21:54
nowenin the database21:54
dystiabut what initiates the log message?21:55
dystialike.  what is it logging the message in response to21:55
nowenwhen the NAS sends it the Accounting packets21:55
nowenis 10.20.2.21 your openvpn server?21:55
dystiano.  :)21:59
dystiait is where ias lives.21:59
nowenso, you need to either stop it from sending packets or archive the logs every month or so22:00
dystiaheheheh22:01
dystiayeah - it can be turned off.22:01
nowendid you archive the rest of the logs successfully?22:01
dystiahonestly i'm not that worried about that part right now?  I am going to have someone go in and fix the cacerts on the two boxes with problems and turn off accoutning on wikid and openvpn22:01
nowenbtw - i'll be on email only tomorrow and very spotty coverage at that22:01
dystiaso that'll reduce volume22:01
dystiawhen I do that i'll have them archive / clean the logs.22:01
dystiathanks.  this has been very helpful.22:02
dystiago tell your dude i buy beer.  plz.  :)22:02
dystiaand the cute.  also single, if that helps.22:02
nowenwell, he's on a different continent, so geographically undesirable may come into play22:02
dystiasiiigh.22:03
nowenalso, I'm guessing his company owns the code22:03
nowenbut maybe you'd move to Jordan?22:03
dystiathat would be an issue.  he should opensource it.  :)22:03
dystiaNo.  not so much.  :)22:03
dystiai'm a little spoiled by being able to walk around on the street w/out a guy - i don't think jourdan is one of the BAD ones, but like.  spoiled american girl.22:03
nowenI hear ya22:03
dystiaaight.  i have a direction.22:04
dystiai'll email you if I need anything tommorow;  i think we'll be ok.22:04
dystiathanks for your help.  :)22:04
nowenmy pleasure22:04
dystiattyl22:04
*** dystia has quit (Quit: Page closed)22:04
*** nowen has quit (Quit: Leaving.)22:22
*** Robinson (be1bfd83@gateway/web/freenode/ip.190.27.253.131) has joined #wikid22:35
Robinsongood afternoon22:35
RobinsonI have a question about licesing22:36
Robinsonanyone here22:37
Robinson????22:37
RobinsonI think no one22:38
Robinsontks22:38
*** Robinson has quit (Client Quit)22:38

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!