| *** nowen (~nowen@99-174-93-102.lightspeed.tukrga.sbcglobal.net) has joined #wikid | 12:52 | |
| *** Tim__ (b899e60b@gateway/web/freenode/ip.184.153.230.11) has joined #wikid | 13:09 | |
| Tim__ | Hi Nick, i got the 2FA working last week. I can get a pass code from both my XP machine and iPad. I have 2 questions though, do you have an application that makes the XP machine require the passcode tpo log in, and also what protocols are used in 2FA on the network? All ive found is challenge/response. | 13:28 |
|---|---|---|
| nowen | Tim: you can try pgina for the windows login | 13:34 |
| nowen | protocols: radius/ldap/saml | 13:34 |
| Tim__ | ok, thanks. Ill look into that right now. | 13:35 |
| nowen | not many people use 2FA for windows login. it is mostly for remote accesss | 13:35 |
| *** palguay (7ab36186@gateway/web/freenode/ip.122.179.97.134) has joined #wikid | 13:37 | |
| Tim__ | ok, what percentage would you say would use it for login? Ive never used it before and was interested and assumed it was used on campus for log ins. | 13:37 |
| Tim__ | no exact figures needed, just rough estimate off the top of your head in case im asked during my presentation, thanks | 13:39 |
| palguay | Hi I have inherited a system running wikid and do not know enough of the system to debug an issue we are having , Can someone point me to the right documents/logs to look at this | 13:39 |
| Tim__ | Nick would be the one to ask, he would need the issued to point you in the right direction though | 13:41 |
| palguay | when a user logs into a system we get a cannot find group id for a logged in user ( linux system) | 13:42 |
| nowen | palguay: what version of WiKID is this? | 13:42 |
| palguay | rpm gives me this wikid-server-enterprise-3.4.1.b3314-1 | 13:43 |
| palguay | it was running fine not sure what changed and started causing this error | 13:44 |
| *** nowen has quit (Remote host closed the connection) | 13:45 | |
| palguay | one thing I noticed is this error in the messages ERROR: permission denied for relation full_domain_keys | 13:45 |
| *** nowen (~nowen@99-174-93-102.lightspeed.tukrga.sbcglobal.net) has joined #wikid | 13:46 | |
| nowen | sorry | 13:47 |
| nowen | back now | 13:47 |
| nowen | so - did the server get restarted and then the issue happened? | 13:50 |
| nowen | check the certificates: http://www.wikidsystems.com/support/wikid-support-center/troubleshooting-faq/how-do-i-know-if-my-certificate-is-valid?searchterm=keytool | 13:51 |
| palguay | when I do a service wikid status I get the Master functions not functioning properly | 13:53 |
| nowen | you can ignore that. you have to edit a file for that to work | 13:54 |
| palguay | looking for the passphrase , seem to have forgotten | 13:57 |
| palguay | give me a few minutes | 13:57 |
| nowen | it might be in /etc/WiKID/security | 13:58 |
| palguay | great I see it there | 13:58 |
| palguay | thanks | 13:58 |
| palguay | the certificate seems valid | 14:01 |
| nowen | and the localhost too? | 14:01 |
| palguay | yes both | 14:01 |
| nowen | ok - set your logging to debug and try to login again: http://www.wikidsystems.com/support/wikid-support-center/troubleshooting-faq/how-can-i-set-radius-logging-to-debug-how-can-i-see-if-wikid-is-getting-the-radius-requests?searchterm=radius+debug | 14:02 |
| nowen | so, the error you're getting is on the system your logging into - after the user has authenticated? | 14:03 |
| palguay | yes after the user has authenticated .. I do not see the Logger file | 14:05 |
| palguay | is is Wikid-syslog.xml | 14:06 |
| nowen | it is in the webgui / WiKIDAdmin | 14:06 |
| palguay | sorry for the noob question, I have not used the GUI before are the credentials the same as what is in security | 14:10 |
| nowen | not necessarily | 14:10 |
| palguay | I have got someone to login and enable that | 14:18 |
| nowen | ok - you'll need access to the WiKIDAdmin for us to troubleshoot effectively | 14:18 |
| nowen | also - are you sure this is a WiKID issue? Is WiKID supposed to send group info via radius attributes? | 14:19 |
| palguay | I am not sure about that . Does this error mean something ERROR: java.io.IOException: PKCS12 key store mac invalid - wrong password or corrupted file. | 14:23 |
| nowen | yes, it means that your certificate is invalid/expired or that the passphrase is incorrect | 14:24 |
| nowen | so - the user is authenticated on the server and then rejected do to a group authorization issue? | 14:24 |
| *** Smithart (1899c122@gateway/web/freenode/ip.24.153.193.34) has joined #wikid | 14:24 | |
| palguay | both the userid and group id cannot be found once the authorization is done | 14:27 |
| nowen | what system is responsible for that? | 14:36 |
| palguay | I am not sure but am assuming that wikid gets this from ldap | 14:38 |
| nowen | that's not possible | 14:38 |
| nowen | most likely your SSH server gets it from ldap | 14:38 |
| nowen | the most WiKID can do it return a radius attribute that provides a group that is then matched on the server | 14:39 |
| palguay | here is something from the wikid radius.log Check PAP bombed with AccessRejectException: Access Denied | 14:42 |
| nowen | that is a user getting denied. you will need the debug log to know why. | 14:43 |
| palguay | there is this error com.wikidsystems.server.wAuth: Database error while validation offline response | 14:44 |
| *** Tim__ has quit (Ping timeout: 245 seconds) | 14:45 | |
| nowen | here's the best way to find out what is going on: set the WiKIDAdmin logs to debug. Try to login - then post the entire logs on pastebin.com | 14:45 |
| palguay | WiKIDAdmin logs from the GUI ? | 14:49 |
| nowen | yes | 14:51 |
| Smithart | com.wikidsystems, com.wikidsystems.radius.log.DBSvrLogImpl, and com.wikidsystems.server.wAuth are all set to debug. The only errors in the GUI are: | 14:51 |
| Smithart | ERROR: java.io.IOException: PKCS12 key store mac invalid - wrong password or corrupted file. | 14:52 |
| Smithart | com.wikidsystems.server.wAuth: Database error while validation offline response | 14:52 |
| nowen | ok - let's update the server to the latest version and create new certs | 14:52 |
| nowen | is this a vm? | 14:53 |
| palguay | no it is not a vm | 14:54 |
| nowen | how do you do back-ups? | 14:54 |
| palguay | our hosting provider does backup - this shows up on mouseover org.postgresql.util.PSQLException: ERROR: permission denied for relation full_domain_keys | 15:01 |
| nowen | can you have your hosting provider do a back up now? I think we should upgrade the server and create new certs. | 15:02 |
| palguay | we have a master slave setup if that can help | 15:03 |
| nowen | yes, that's good | 15:03 |
| nowen | we will have to upgrade the master and then upgrade the slave | 15:03 |
| nowen | here are the two rpms: http://wikidsystems-dl.com/wikid-server-enterprise-3.5.0.b1428-1.noarch.rpm | 15:04 |
| nowen | and http://wikidsystems-dl.com/wikid-utilities-3.4.3-1.i386.rpm | 15:04 |
| nowen | I assume this is a 32 bit server? | 15:04 |
| palguay | this is a 64 bit server | 15:06 |
| nowen | scratch that last one then | 15:06 |
| palguay | Is there a 64 bit rpm for the utilities ? | 15:11 |
| Smithart | how risky is this? these servers are the gateway to (and between) our production servers. | 15:11 |
| nowen | bulding it now | 15:11 |
| Smithart | we have other means to authenticate, but i just want to make sure it won't indirectly break those | 15:12 |
| nowen | how many WiKID users do you have? it will be on the home tab | 15:12 |
| Smithart | Licenses In Use:4 | 15:12 |
| Smithart | i don't know how many we bought; probably 10? | 15:13 |
| nowen | yeah, that's the min. I just wanted to know what we're dealing with | 15:13 |
| Smithart | that server is our bastion server, and it proxies ssh connections from the outside into our secure servers | 15:14 |
| Smithart | and in some cases between internal servers | 15:14 |
| Smithart | wikid is our secure way in when we can authenticate into the environment thru that server using keys | 15:15 |
| Smithart | can't* | 15:15 |
| nowen | gotcha | 15:15 |
| nowen | http://wikidsystems-dl.com/wikid-utilities-3.4.3-1.x86_64.rpm | 15:24 |
| nowen | so, we have a new certificate in the latest rpm - so if we are going to do new certs, we need to update the rpm. | 15:25 |
| nowen | I don't think it is too risky, but you never know. it sounds like you also run other services on the server | 15:25 |
| nowen | stop wikid and then run 'killall -9 java' | 15:28 |
| nowen | to make sure everything has stopped | 15:28 |
| nowen | then, do 'rpm -Uvh wikid...' on those rpms. the utilities first | 15:28 |
| palguay | failed dependancies on the utilities | 15:34 |
| palguay | rpmlib(FileDigests) <= 4.6.0-1 is needed by wikid-utilities-3.4.3-1.x86_64 rpmlib(PayloadIsXz) <= 5.2-1 is needed by wikid-utilities-3.4.3-1.x86_64 | 15:34 |
| palguay | we are on redhat EL 5.6 | 15:38 |
| nowen | if you do 'yum update --nogpg wikid-*' does it list the reqs? | 15:39 |
| palguay | yes it seems to resolve dependancies , does not list the payload and filedigest | 15:43 |
| nowen | good | 15:43 |
| palguay | you want me to update with yum ? | 15:44 |
| nowen | yes | 15:44 |
| palguay | got the same error when it tries to rpm_check_debug | 15:46 |
| nowen | sorry - did the yum update not work? | 15:46 |
| palguay | no it failed with the same error | 15:48 |
| palguay | ERROR with rpm_check_debug vs depsolve: rpmlib(FileDigests) is needed by wikid-utilities-3.4.3-1.x86_64 rpmlib(PayloadIsXz) is needed by wikid-utilities-3.4.3-1.x86_64 | 15:49 |
| nowen | hmm. works for me on el5 centos box | 15:50 |
| nowen | did you use the --nogpg? | 15:51 |
| nowen | what version of rpm is on this system? | 15:52 |
| palguay | yes I used that | 15:52 |
| palguay | RPM version 4.4.2.3 | 15:53 |
| nowen | ok hold on. | 15:55 |
| nowen | I will have to rebuild | 15:55 |
| nowen | ok - try the same link again for the utilities rpm | 16:03 |
| palguay | the utilities package installed :-) | 16:04 |
| nowen | yay! | 16:04 |
| palguay | Is the server package the same ? | 16:05 |
| nowen | yes, use the same command | 16:05 |
| palguay | ok the installation completed | 16:09 |
| nowen | ok - restart wikid | 16:09 |
| palguay | service wikid restart Restarting WiKID sudo: sorry, a password is required to run sudo Tomcat server already stopped. TimeCop process already stopped. Logger process already stopped. Database already stopped. ssh_exchange_identification: Connection closed by remote host rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7] Synchronizing master fi | 16:11 |
| nowen | did you setup wikid as a service? | 16:11 |
| palguay | yes it was already setup | 16:11 |
| nowen | hmm, it sounds like something with the slave | 16:12 |
| palguay | ok I got logged out of the system , not sure why | 16:12 |
| nowen | did it start? | 16:16 |
| palguay | we seem to be locked out of our systems | 16:17 |
| nowen | hmm, I don't think we did anything that would cause that | 16:17 |
| palguay | we are trying to see if we can get in someplace | 16:21 |
| Smithart | we both got booted from the server when wikid restarted | 16:21 |
| nowen | hmm, the command to restart starts postgres and tomcat | 16:21 |
| nowen | can you tell if the server is up at all? | 16:22 |
| *** palguay_ (7ab36186@gateway/web/freenode/ip.122.179.97.134) has joined #wikid | 16:23 | |
| palguay_ | I can ping the server | 16:23 |
| nowen | is ssh up on port 22? | 16:24 |
| *** Smithart has quit (Ping timeout: 245 seconds) | 16:25 | |
| *** palguay has quit (Ping timeout: 245 seconds) | 16:25 | |
| palguay_ | yes ssh seems to be up | 16:25 |
| *** Smithart (1899c122@gateway/web/freenode/ip.24.153.193.34) has joined #wikid | 16:25 | |
| nowen | but you cannot login? | 16:25 |
| Smithart | no. we can still log into the slave | 16:26 |
| Smithart | but not the wikid master server | 16:27 |
| nowen | can you login to the master from the slave? | 16:27 |
| palguay_ | ok I am able to get in | 16:27 |
| Smithart | me too, apparently it was just really slow to come up? | 16:27 |
| palguay_ | only on process for wikid /opt/WiKID/bin/usogres | 16:29 |
| nowen | palguay_: ? | 16:30 |
| nowen | if you run 'netstat -anp | grep 444' is there a listener? | 16:30 |
| palguay_ | netstat -anp |grep 444 tcp 0 0 127.0.0.1:2514 0.0.0.0:* LISTEN 4443/spiped tcp 0 0 192.168.60.189:35419 192.168.60.191:22514 ESTABLISHED 4443/spiped tcp 0 0 127.0.0.1:2514 127.0.0.1:40564 ESTABLISHED 4443/spiped | 16:31 |
| nowen | sorry - 443 | 16:31 |
| palguay_ | tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8737/httpd | 16:31 |
| nowen | is apache running on this server?> | 16:32 |
| palguay_ | apache is running but the wikid url returns a service temporarily unavailable | 16:33 |
| nowen | yeah, it should be running on 443 but can't. did you have the WiKIDAdmin running on a different port? | 16:33 |
| nowen | are you using apache? | 16:33 |
| Smithart | i suspect wikid was running on a different port | 16:34 |
| nowen | look in /etc/httpd/conf/httpd.conf | 16:34 |
| nowen | there should be a redirect | 16:34 |
| palguay_ | I get this when I check if the service is up | 16:34 |
| palguay_ | sudo service wikid status sudo: sorry, a password is required to run sudo Stopped: WiKID master services not running on localhost. | 16:35 |
| nowen | for status to work, you need to edit a file | 16:35 |
| nowen | it is easier to run netstat against the ports | 16:35 |
| Smithart | based on the httpd conf, it looks like wikid gui should be accessable on 443 | 16:36 |
| nowen | what does it say? | 16:36 |
| palguay_ | These errors show up in wikid error logs (apache) | 16:37 |
| palguay_ | (111)Connection refused: proxy: AJP: attempt to connect to 127.0.0.1:8009 (localhost) failed | 16:37 |
| nowen | Smithart: what does you httpd.conf say? because you cannot have apache and tomcat listening on the same port | 16:37 |
| palguay_ | we have a ProxyPass /wikid/ ajp://localhost:8009/wikid/ | 16:39 |
| nowen | ok - that's for the OTPs | 16:39 |
| nowen | they would use port 80, but you have switched it to 8009 | 16:39 |
| nowen | anything for WiKIDAdmin? | 16:40 |
| palguay_ | we have a redirect for wikidadmin | 16:40 |
| nowen | to what port? | 16:40 |
| palguay_ | the redirect is for http to point to https | 16:42 |
| nowen | look in /opt/WiKID/tomcat/conf/ | 16:42 |
| nowen | do you see more than one server.xml ? | 16:42 |
| nowen | perhaps a server.xml.rpm | 16:42 |
| palguay_ | there is server.xml at /opt/WiKID/tomcat/conf | 16:43 |
| nowen | when you went to the WiKIDAdmin in the past, did you have to append a different port number? | 16:44 |
| palguay_ | no we did not have to append a different port | 16:47 |
| Smithart | https://wikid.genares.net/WiKIDAdmin | 16:47 |
| nowen | hmm. well, I'm not sure what is going on. apache and tomcat can't use the same port | 16:47 |
| palguay_ | there is this line <Connector port="80" enableLookups="false" redirectPort="443" acceptCount="100" debug="0" connectionTimeout="6000"/> | 16:51 |
| palguay_ | wikid was not able to start | 16:53 |
| Smithart | the wikiadmin was definitely working before the upgrade | 16:57 |
| nowen | do you see this line: <Connector port="443" protocol="HTTP/1.1" SSLEnabled="true" | 16:58 |
| palguay_ | Yes that seems to be there in the server.xml | 17:00 |
| palguay_ | <Connector port="443" protocol="HTTP/1.1" SSLEnabled="true" | 17:00 |
| nowen | change that to 8443 from 443 | 17:01 |
| nowen | :q | 17:03 |
| palguay_ | ok changed | 17:03 |
| nowen | are there any errors in /opt/WiKID/tomcat/logs/catalina.out? | 17:03 |
| nowen | towards the end | 17:04 |
| palguay_ | looks like errors were before the upgrade | 17:04 |
| nowen | ok | 17:05 |
| nowen | are you running anything else on this server that uses java? | 17:05 |
| palguay_ | does not look like there is anything that uses java | 17:06 |
| nowen | run 'killall -\9 java' and 'rm /opt/WiKID/tomcat/logs/catalina.out' | 17:06 |
| nowen | then start wikid. | 17:07 |
| nowen | you can tail catalina.out if you like | 17:07 |
| palguay_ | start wikid thorugh service ? | 17:08 |
| nowen | use 'wikidctl start' | 17:08 |
| nowen | just in case there's a bug with the service sripts | 17:08 |
| palguay_ | as root or as user wikid | 17:09 |
| nowen | root should be fine | 17:09 |
| palguay_ | it is asking me for the root password | 17:12 |
| palguay_ | seems to be starting slowly | 17:13 |
| nowen | how much memory is on this machine? | 17:14 |
| palguay_ | ok it seems to have come up | 17:15 |
| palguay_ | around 15 G | 17:18 |
| nowen | well, should be fast then | 17:18 |
| palguay_ | since we changed the tomcat port do we use a port to login to WiKIDAdmin | 17:19 |
| nowen | yes | 17:19 |
| palguay_ | we may have a fire wall issue , checking | 17:21 |
| nowen | perhaps the redirect occurred at the firewall? | 17:25 |
| *** Smithart has quit (Ping timeout: 245 seconds) | 17:26 | |
| palguay_ | maybe | 17:28 |
| palguay_ | I am thinking of changing it back to what it was and restart wikid | 17:28 |
| nowen | sure, then look at catalina.out for an error | 17:28 |
| palguay_ | firewall rules are at our hosting provider | 17:28 |
| nowen | is the person who set it it up originally available? | 17:29 |
| palguay_ | no we are not able to get hold of him | 17:30 |
| palguay_ | let me try to change it back and restart | 17:31 |
| nowen | ok | 17:34 |
| nowen | tail the catalina.out file | 17:35 |
| palguay_ | nothing in catalina.out but there are some errors in catalina.err | 17:44 |
| nowen | what? and is tomcat listening on 443? | 17:44 |
| palguay_ | no we get back the old server unavailable error | 17:50 |
| nowen | what are the errors in catalina.err? | 17:50 |
| palguay_ | http://pastebin.com/Xkc4nGA4 | 17:53 |
| nowen | java.net.BindException: Address already in use <null>:443 | 17:55 |
| palguay_ | yes , when we did an install does it overwrite the old files in /opt/WiKID dir ? | 17:55 |
| nowen | yes | 17:55 |
| nowen | in your server.xml, I recommend you comment out the 80 to 443 redirect and put in: | 17:56 |
| nowen | <Connector port="8090" protocol="HTTP/1.1" | 17:56 |
| nowen | connectionTimeout="20000" | 17:56 |
| nowen | /> | 17:56 |
| palguay_ | let me check the settings on the slave | 17:57 |
| nowen | then, change the 443 port to 8443. | 17:57 |
| nowen | are you using apache on this server? | 17:57 |
| palguay_ | no only for this | 17:58 |
| palguay_ | do you think we can only run tomcat and use standard ports ? | 18:00 |
| nowen | so if you turn off apache, nothing bad happens? | 18:00 |
| palguay_ | yes | 18:01 |
| nowen | then turn it off | 18:01 |
| nowen | we will use the standard ports | 18:01 |
| nowen | why was apache running? | 18:01 |
| palguay_ | not sure | 18:06 |
| nowen | run 'chkconfig httpd off'. It might have just come on after a reboot | 18:06 |
| palguay_ | errors here http://pastebin.com/WJ4JXQKf | 18:24 |
| nowen | run 'ls -all /opt/WiKID/tomcat' | 18:25 |
| palguay_ | drwxr-xr-x 8 wikid root 4096 Apr 29 11:08 . drwxr-xr-x 14 wikid root 4096 Apr 23 21:11 .. drwxr-xr-x 2 wikid root 4096 Apr 29 11:08 bin drwxr-xr-x 3 wikid root 4096 Apr 29 13:18 conf drwxr-xr-x 2 wikid root 4096 Apr 29 11:06 lib -rw-r--r-- 1 wikid root 57846 Apr 23 21:11 LICENSE drwxr-xr-x 2 wikid root 4096 Apr 29 13:16 logs -rw-r--r-- 1 wikid root 1228 Apr 23 21:11 NOTICE -rw-r--r-- 1 wikid root 9054 Apr 23 21:11 RE | 18:27 |
| nowen | run 'netstat -anp | grep 443' | 18:28 |
| palguay_ | there is process listening on 443 | 18:29 |
| nowen | is it jsvc? | 18:29 |
| palguay_ | yes jsvs.exec | 18:29 |
| nowen | can you get the WiKIDAdmin? | 18:29 |
| palguay_ | no | 18:29 |
| nowen | is there an error? | 18:30 |
| palguay_ | connection to the server was reset whie page was loading | 18:31 |
| nowen | was that the entire output of that ls command? | 18:31 |
| palguay_ | http://pastebin.com/9eQXEZSD | 18:32 |
| nowen | odd, there should be a work directory | 18:33 |
| nowen | what user are you? | 18:35 |
| nowen | root? | 18:35 |
| palguay_ | yes | 18:36 |
| nowen | will you run wikidctl setup and then restart? | 18:37 |
| palguay_ | ok | 18:39 |
| palguay_ | looks like networking is already configured | 18:40 |
| nowen | just step through it anyway | 18:40 |
| palguay_ | yes or no | 18:41 |
| nowen | yes | 18:41 |
| palguay_ | do I use both eth0 and eth1 | 18:44 |
| nowen | I don't know, one is typically used for the external IP and one for the internal. I recommend you ctrl-c out of this rather than risk messing up your network | 18:46 |
| palguay_ | let me check something | 18:48 |
| nowen | also, can you run 'ls -all /opt/WiKID/tomcat/logs' | 18:51 |
| palguay_ | http://pastebin.com/WCxX1u0y | 18:56 |
| nowen | palguay_: did you check on your thing? | 19:36 |
| palguay_ | yes | 19:44 |
| nowen | and? | 19:44 |
| palguay_ | there seem to be too many things around wikid that have been setup and it might be of not much use to go forward till that is figured out | 19:45 |
| nowen | you mean to many customizations or too many other things running on the server? | 19:45 |
| palguay_ | yes customizations and security setup | 19:45 |
| nowen | do you want to start over with a fresh setup? | 19:46 |
| palguay_ | not sure about that now | 19:47 |
| palguay_ | I will ping you once we decide how to go about this | 19:52 |
| nowen | ok | 19:52 |
| nowen | also, I need to send you'll an invoice. | 19:52 |
| nowen | is there a good email for that? | 19:54 |
| nowen | srane? | 19:55 |
| palguay_ | yes | 20:01 |
| nowen | thx | 20:01 |
| *** palguay_ has quit (Ping timeout: 245 seconds) | 20:41 | |
| *** nowen has quit (Quit: Leaving.) | 21:06 | |
| *** nowen (~nowen@99-174-93-102.lightspeed.tukrga.sbcglobal.net) has joined #wikid | 21:07 | |
| *** nowen has quit (Read error: Connection reset by peer) | 21:17 | |
| *** nowen (~nowen@99-174-93-102.lightspeed.tukrga.sbcglobal.net) has joined #wikid | 21:18 | |
| *** nowen has quit (Quit: Leaving.) | 22:10 | |
Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!