Root Access

Jauyzed

5:07 AM

@Bob hello

Bob

@Jauyzed hi!

(context for others: superuser.com/questions/1218694/…)

Jauyzed

alright cool. I wasnt sure if you were same bob...haha. alright now that we are on the context

Bob

@Jauyzed Just to keep things simple - first, let's make sure you can access your home from the EC2 instance. You mentioned you tried curl -- I see a redirect. Try curl -L (-L for follow redirects)?

Jauyzed

curl -L <domainname> from EC2 instance, i get a connection refused on 443

i have those ports open in the PF sense and also in the sec groups of EC2 instance

Bob

@Jauyzed So, right now we are trying to verify that EC2 can load the website on your home IP.

is <domainname> there pointing to the EC2 instance or to your home?

Jauyzed

5:21 AM

@bob domain name --> EC2 instance --> home ip

dig points to ec2 instance

ip address

Bob

@Jauyzed Yes. The question right now is, if you execute that curl -L command from the EC2 instance, targeting your home address, does the correct page come back?

Right now, we don't care about external access. Start back at step 1. Make sure EC2 has access to your home address.

Jauyzed

no

Bob

@Jauyzed What do you get instead?

Jauyzed

i get a connection refused

on 443

since i have a cert in the home nginx server with a redirect from 80 to 443:-)

Bob

hm. dammit it's probably getting redirected to itself

Ok, let's start in the other direction.

@Jauyzed run netstat -anpt4 on EC2

paste the output here if you can, let's see what's listening

Jauyzed

5:27 AM

i did a curl -L <homeip> from EC2 instance. i get curl: (51) SSL: certificate subject name (mydomainname) does not match target host name '<homeip>'

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 172.31.69.29:80 54.227.108.186:48490 TIME_WAIT -
tcp 0 604 172.31.69.29:22 <homeip>:55292 ESTABLISHED -

Bob

@Jauyzed ok, there's your first problem: your EC2 nginx instance isn't listening on 443

Jauyzed

thats the output of netstat -anpt4

Bob

you're going to have to set up HTTPS on the EC2 nginx instance

If the target server redirects from 80 to 443, that redirect is actually processed by the client. Which will then try to hit 443 on EC2 and ... fail. Because it's not listening.

Jauyzed

i have to add the ssl block in the reverse proxy ?

Bob

Yup.

Jauyzed

5:31 AM

gotcha

Bob

Don't do the redirect there, just make sure it's listening on ssl (and also has the certificates installed)

Jauyzed

let me give that a shot.

Bob

@Jauyzed Basically, you want to add your listen 443 ssl; and all the ssl_ directives in the same server block that 80 is on.

server {
    listen 80;
    listen 443 ssl;

    ssl_certificate ...;
    ...

    location / {
        proxy_pass ...;
        ...
    }
}

and your proxy_pass should be proxy_pass $scheme://blah

(alternatively, you could always use https between ec2 and home, but then you'd need to put your 80 => 443 redirect on the ec2 side and only do the reverse proxy on https)

Jauyzed

i think ^ is better

Bob

hm, actually, ignore that bit about $scheme... apparently that doesn't work O_O

Jauyzed

5:40 AM

let me comment out the ssl part in the homeserver to check if reverse proxy works on 80

Bob

@Jauyzed Yea, the only way might be to always use https in your proxy_pass, and define a separate listen 80 that does the redirect on ec2

@Jauyzed you would need to remove the redirect

hm. I can't remember how I used to do this :P

oh, that's right. backend was http only (cause they lived on the same physical server)

Jauyzed

Ok. if i hear you right, do the redirect on ec2 instance and have only 443 open on home server

Bob

@Jauyzed yup

Jauyzed

cool. let me do that tonight. if i hit a wall...i will reply back on the question. Thanks for responding :-)

Bob

@Jauyzed Good luck!

Hopefully it's not an AWS issue, cause configuring that is a pain :P

satibel

5:49 AM

@Rahul2001 yay

Rahul2001

@satibel bloody hell. Why isn't the sandbox working?

Jauyzed

@bob Thanks! I will keep you posted.

Bob

lol

@Rahul2001 mind sharing the source? :P

Rahul2001

@Bob fixed :P

Bob

@Rahul2001 still, mind sharing? I'm curious :P

Rahul2001

5:56 AM

@Bob Here you go:

Bob

@Rahul2001 oof, I didn't mean paste it here :P

Rahul2001

The array $na contains files which I do not want displayed

@Bob Meh, doesn't matter

Bob

might wanna remove it. public google-indexed and all :P

uhm

Rahul2001

Got it?

Bob

@Rahul2001 I think you need a proper sandbox. Not funny string manip. See: hackchal.cf/proxy.php?url=/etc/passwd

Rahul2001

5:59 AM

@Bob man

Bob

lol

@Rahul2001 I'm thinking something chroot

Rahul2001

I think for now I'll just be replacing sensitive strings with foo. Can't do much, limited privileges on webhost

Bob

@Rahul2001 where are you hosting it?

Rahul2001

Will (hopefully) get a decent server if I can get hold of sponsors for the competition (as a whole, which includes tons of other events too)

@Bob gridhoster.com

Bob

oh

@Rahul2001 My offer of a server is still open if you want it :P

Avery

6:01 AM

do is mine.

Rahul2001

Huh, it'll probably get abused pretty bad by the participants...

You guys up for that?

Avery

maybe

Bob

@Rahul2001 Sure. It's a disposable box anyway :P

lxc delete foo and it's all clean

(and if they do manage to break out ... meh, lxd stress-testing)

@Rahul2001 when/how long does this event run for?

Rahul2001

@Bob August/October - For a week

Bob

oh, a week is easy enough

Rahul2001

6:06 AM

@Bob huh ok, but I'm still going to try and get sponsors because I want a server for myself anyway :P

If I'm not able to, I'll come running to you :P

Bob

*shrug* up to you. If you want it now for testing and then migrate off later, that's fine too :P

I don't mind you using it... asking about the time was more to know what times I shouldn't mess around with the host so I don't bring it all down at a bad time :P

Hm... though... I could chuck you on the other, more stable, server

Rahul2001

I'll let you know :)

Bob

@Rahul2001 ~3EUR/month for a decently good server. $1/month for a really cheap one :P

Rahul2001

@Bob Huh, where?

Bob

dunno where you'd look for sponsors though

@Rahul2001 really cheap? lowendbox.com. decently good? scaleway.com

djsmiley2k

6:10 AM

my god I'm so tired this morning :(

Bob

(I say decently in that they're somewhat professional - but they're not the most, ah, reliable)

Jauyzed

@bob i get a 502 bad gateway

for my setup

Bob

@Jauyzed progress! :P

Jauyzed

dang it

Bob

@Jauyzed Is that when you access the ec2 one?

That usually means the upstream host is down

(or inaccessible)

Jauyzed

6:12 AM

just from browser

Bob

@Jauyzed yea, but are you accessing ec2 or home directly?

Jauyzed

the upstream is working.

Bob

ok

@Jauyzed is upstream listening on 443 only now?

what's your proxy_pass line?

Jauyzed

just added $schema://

actually both servers have 80 and 443 open

with certs installed

Bob

@Jauyzed use https:// instead, apparently $scheme doesn't work there

Jauyzed

6:16 AM

im redirecting only in the EC2 instance

Bob

proxy_pass https://...

Jauyzed

yeap

Bob

@Jauyzed does it still 502?

Jauyzed

ya :-(

Bob

@JourneymanGeek github.com/n1trux/awesome-sysadmin#it-asset-management

hmm

@Jauyzed can you provide the full configs?

Jauyzed

6:24 AM

EC2 nginx.conf

worker_processes 1;

events { worker_connections 1024; }

http {

sendfile on;

upstream docker-nginx {
server <homeip>;
# access_log /var/log/nginx/access.log;
# error_log /var/log/nginx/error.log;

}

server {
listen 80 default_server;
listen [::]:80 default_server;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;

ssl on;
ssl_certificate /etc/nginx/ssl/bundle.crt;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;

Bob

hmmm

I feel kinda dumb for not having asked earlier, but... error logs, @Jauyzed?

Jauyzed

only diff on my home nginx is i using sites-enabled...i have a conf there

server {
listen 80 default_server;
listen [::]:80 default_server;

#return 301 https://$host$request_uri;
}
# SSL configuration
#
server {
#listen 443 ssl;
#listen [::]:443 default_server ssl;

listen 443 ssl http2;
listen [::]:443 ssl http2;

....

Bob

@Jauyzed check error logs, and just for kicks try turning off http2 for now

FYI no http2 with reverse proxies for now anyway serverfault.com/questions/765258/…

(as in it should fall back to 1.1... but turn it off while testing for now)

Jauyzed

server {
listen 443 ssl;
listen [::]:443 ssl;

removed http2

Bob

oh

@Jauyzed did you restart nginx after changing the configs? :P

Jauyzed

6:35 AM

yep

doing it everytime

Bob

heh, I miss that a lot

Jauyzed

;-)

Bob

@Jauyzed did you remove it on both servers?

Jauyzed

yep

still the same 502

Bob

hmmmmm

@Jauyzed anything in error logs?

/var/log/nginx/error.log

Nick

6:37 AM

SHAMELESS SELF PROMOTION:

My Actual First Video

►

^ My first youtube channel

Please comment. Do not click the dislike button even if you dislike it. You are free to insult me in the comments. Thank you :)

*dislikes*

NOOO!!!

:P

XD

can't actually load yt at work

well, can. but shouldn't.

Nick

6:39 AM

awesome. you won't like it anyway, it's me pretending to be an alien indexing a human teenager's room.

djsmiley2k

@Nick You realise a Dislike is actually better for your channel than nothing at all?

Nick

To all the awesome people who do watch it and say something about it, thank you.

Bob

@Jauyzed Also, which Amazon image did you use?

Jauyzed

@bob SSL_do_handshake() failed (SSL: error:140770FC

thats the error log on ec2 instance

Bob

o.O

Jauyzed

6:41 AM

exactly

using the same certs

Bob

@Jauyzed anything else?

tbh I was expecting a selinux-related permission error :P

Jauyzed

im using ubuntu 16

Bob

(just checking, you're looking at the bottom of error.log? try tail /var/log/nginx/error.log)

Jauyzed

i see the whole error log in 20 lines

Bob

that's a remarkably short error log

Nick

6:44 AM

@djsmiley2k yeah, but I prefer that people comment. I like to hear words, not binary likes and dislikes.

Bob

@Jauyzed does it add an error every time you try to access it?

it should :P

every 502 should have an error attached

Jauyzed

@Bob i have access logs

Nick

@Bob mine was less

Bob

@Jauyzed but no errors corresponding to the 502 hits?

Jauyzed

172.58.62.213 - - [14/Jun/2017:06:33:23 +0000] "GET / HTTP/1.1" 301 194 "-" "curl/7.54.0"
172.58.62.213 - - [14/Jun/2017:06:33:55 +0000] "GET / HTTP/1.1" 301 194 "-" "curl/7.54.0"
172.58.62.213 - - [14/Jun/2017:06:33:55 +0000] "GET / HTTP/1.1" 502 182 "-" "curl/7.54.0"

thats the access log on ec2 instance

Bob

6:46 AM

@Jauyzed why is it curl? I thought you were testing with a web browser?

Jauyzed

tried both

14/Jun/2017:06:32:40 +0000] "GET / HTTP/1.1" 502 584 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

that my browser

Bob

Ok. Do the error timestamps correspond with these accesses?

Jauyzed

yes

Bob

@Jauyzed can you post the whole error line?

not just the beginning bit

Jauyzed

here is the tricky part...im sitting on my home ip

Bob

6:49 AM

In theory, that should not be a problem.

Jauyzed

[14/Jun/2017:06:50:35 +0000] "GET / HTTP/1.1" 502 584 "-" "Mozilla/5.0 (Linux; Android 6.0.1; SM-G900T Build/MMB29M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.83 Mobile Safari/537.36"

Bob

@Jauyzed The error line, not access

Jauyzed

thats from my cellphone

Bob

11 mins ago, by Jauyzed

@bob SSL_do_handshake() failed (SSL: error:140770FC

Where's the rest of the error?

Even better if you can just dump the entire error.log

Jauyzed

2017/06/14 06:49:51 [error] 11470#11470: *36 SSL_do_handshake() failed (SSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) while SSL handshaking to upstream, client: 172.58.62.213, server: , request: "GET / HTTP/1.1", upstream: "https://<homeip>:80/", host: "mydomainname"

Bob

6:54 AM

hm...

wait.

https://<homeip>:80/

why is it :80

it should be 443

@Jauyzed Are you specifying the port in your upstream / server directive?

try specifying 443, either in the upstream or in the proxy_pass

Jauyzed

not specifying 80 or 443

Bob

@Jauyzed looks like it's defaulting to 80 anyway... add a :443

Jauyzed

i just did

let me restart and try it

yeap

it works now

Bob

\o/

Jauyzed

reverse proxy complete

Bob

6:59 AM

@Jauyzed you might want to add that http2 back

and make sure the 80 to 443 redirect works

last step would be to toss it through ssllabs.com/ssltest and see how it performs :P

Rahul2001

'lo

I'm at the cyber cafe again, this time to download arch linux

Jauyzed

@bob ssl is good

Rahul2001

WHOA

800kB/s torrenting \o/ :P

Jauyzed

@bob Thank you.

@bob im going to split now...i can add the answer to my question quoting you unless u want to add it yourself ;-)

Rahul2001

@jokerdino Check this out! IIT mirror has many popular distros mirror.cse.iitk.ac.in

Also ftp.iitm.ac.in

Man, what's up with everyone in my family?

jokerdino

7:11 AM

@Rahul2001 thank you

K is kanpur?

M is what?

Madras?

Rahul2001

@jokerdino yeah k=kanpur

Bob

@Jauyzed I'm writing up the more detailed troubleshooting :P

Rahul2001

@jokerdino yeah, appears to be be so

djsmiley2k

You guys ever heard of a UTR setting?

I'm not even sure what it's related

Bob

@djsmiley2k lmgtfy gov.uk/limited-company-formation/…

needs more context though :P

Rahul2001

7:15 AM

back home!

Bob

@Rahul2001 that was quick

is the cafe like next door? :P

jokerdino

what happened last night?

Rahul2001

@Bob no, as in, on my way back home

@jokerdino whoooooooooo. So, what's the weather like?

:38100949 ;P

WHY U DELETE THAT!?

Bob

@Rahul2001 finger slipped

Rahul2001

lol

man, why did arch get copied so fast but ubuntu is taking ages?

There's a hot girl at the cafe... To bad that I'm leaving this city and going back home in a couple of days...

jokerdino

7:26 AM

wut

good thing you are leaving the poor girl alone?

Rahul2001

@jokerdino @_@

Bob

@Rahul2001 niltalk.com

lol

github.com/knadh/niltalk

Rahul2001

@Bob NOOOOOOOOOOOOO!

Heh, it's overly complicated

No support for text formatting

Mine is better, MUCH BETTER

NO NO NO SHUTUP MINE IS BETTER AND IT RUNS ON PHP

jokerdino

wtf

Rahul2001

Everyone knows that PHP is the most superior

Bob

7:33 AM

!!no

:D

hm

jokerdino

ignored

Bob

*poke*

!!info

hmmmmmmmmmmmmm

jokerdino

##info

I Am A Bot

@jokerdino I awoke on Wed, 14 Jun 2017 00:02:46 GMT (that's about 7 hours ago), got invoked 5 times

jokerdino

##no

I Am A Bot

7:35 AM

Bob

fork(): Operation not permitted
fork(): Operation not permitted
04:52:31.154 INFO - Session dccc89e5-840c-4618-8dfa-a8d183665831 deleted due to client timeout
1497415951163   Marionette      INFO    New connections will no longer be accepted
[GFX1-]: Receive IPC close with reason=AbnormalShutdow[Child 780]n WARNING: pipe error
(31): Connection reset by peer: file /home/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[Child 780] WARNING: pipe error (3): Connection reset by peer: file /home/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_…

that no look good

!!info

ChatBot John Cavil

@Bob I awoke on Wed, 14 Jun 2017 07:37:58 GMT (that's about 19 seconds ago), haven't done anything yet!

Bob

@allquixotic firefox (nightly) still rather unstable... it randomly crashed a couple hours ago

maybe in ~2 months when this moves to release :P

jokerdino

can we get bot wars?

Bob

@jokerdino no.

jokerdino

7:39 AM

:(

Rahul2001

YEAH Y NOT

@Bob what version are you using?

Meh, if someone offers me 200 rep, I'll create a bot in pure batch with curl :P

Bob

@Rahul2001 most recent nightly

Rahul2001

Just to prove the levels of hackery I can reach

Bob

I would have to kill it. With fire.

Rahul2001

@Bob headless?

Bob

7:44 AM

@Rahul2001 ya?

Rahul2001

@Bob hackery is fun

@Bob I didn't know that existed

Bob

@Rahul2001 there's a reason it's nightly

djsmiley2k

@Bob in networks, it's appently what's caused our problem for the last 2 days with this client.

I'm everyone's response on the desk is 'what's UTR?'

I may of covered it in CCNA but that was like 10 years ago.

Bob

@djsmiley2k unwanted traffic removal?

djsmiley2k

orly?

Bob

7:49 AM

team-cymru.org/UTRS

Burgi

morning

djsmiley2k

Don't think it's that @Bob :/

I'm wondering if they meant URT

sigh

tightvnc doesn't have an option to connect and take a screenshot

I was just about to do something awesome at work, but defeat!

satibel

my morning: psu is crapped out, fetch another psu from the "crap but working PCs" pile
try to debug why the pc still isn't booting for an hour, changing mobos, ram, gpu etc... still no go... then the "new" psu goes bang, fetch the last psu, works well, but the fan's bearing are dead, need to get a new 120mm fan, none of the dead psus have one, I finally found one on a pc, open the psu, need to change the connector because of polarity issues, finally put it back together and I'm here.

djsmiley2k

well done

Transcript for