@AndroidDev Some languages have more than one regex dialect. :) Anyway, I had meant to mention earlier that this site is pretty useful as a regex tutorial and reference.
I have some experience with linux having played with raspberry Pis a bit but I am still a relative noob...
I am setting up a pc running XUbuntu and so far it is going pretty well. I am having issues getting wifi working though.
Initially I tried with my existing Netgear A6200 but was told tha...
I installed Linux Mint 18 on a separate partition on my PC.
In order to post threads about any problem that I will encounter:
Can I do so in Ask Ubuntu?
Or in the Linux Mint forums?
Am I running an Ubuntu server on AWS as the web service section of my mobile game Quote Cipher. I have googled to try to figure this out, but I'm either not phrasing it right or this question just isn't out there.
Every once in a while, my server simply quits responding to my game. The first few...
@ByteCommander Have you figured out what's going on? I'm thinking of posting on Unix & Linux about the issues that are blocking me from being confident in my answer to your cat | read question. Based on my reasoning, cat | read -rdz should terminate after a z + newline-or-flush. But it doesn't. I've tried in bash, ksh93, and zsh in case the shell mattered. (I can't try shells like dash as they don't support read -d.)
@WinEunuuchs2Unix : Very much on purpose, I only have a modem from my ISP. I want my Ubuntu server that is my gateway/firewall/router to be direct connected to the WAN. Why? Because I study this stuff, meaning hacking attempts and such and specifically do not want my ISP device to be a modem and router, but just a modem.
@WinEunuuchs2Unix : I have an idea to greatly (by at least 100X, probably even more) increase the speed of the bytes per IP address per unit time portion of the work flow. However, it will take some programming. If your vnstat method is better or preferred, then maybe I won't bother just now.
@DougSmythies vnstat doesn't increase speed, it simply records amount of traffic coming in and going out. It doesn't log traffic by IP address though. I only figured out which IP site by closing down all other websites in chrome and/or firefox. I just did testing under Kernel 4.12.10 and upload damage isn't as bad with flashplayer 25% tx/rx ratio instead of 186% under kernel 4.4.0-93. I'll update the answer now.
I have a setup on ubuntu and 3 external HD are connected to it. I used the command ls /sys/bus/usb/devices which shows me I have 2 usb controllers usb1 and usb2. Now I need to know how many free slots are available under both the controllers so that I know how many more external HD can be attache...
@DougSmythies I'm not sure which part is dreadfully slow... The C or the Dash or the tcpdump? The bigger stumbling block for me and others like me is being gcc (c compiler) challenged. Is tcpdump running all the time as a daemon?
@WinEunuuchs2Unix : Currently the dreadfully slow part is the grep per IP address part. The idea is to just sort the entire file, based on field 13, and then process it once instead of the 5000 times I did. Perhaps I ask my own question to see if someone has some great idea how to do that step without a c program. I never run tcpdump as a daemon, but yes, in the example I gave, it runs all the time, changing output files every hour.
@WinEunuuchs2Unix : For what I do, I change files every 10 minutes to keep them a manageable size for times when I come back and look at them with wireshark.
@DougSmythies I try to do everything with bash and avoid things like grep, sed and awk. As such my initial design would be to load everything into an array and perhaps an associated array of keys (IP address + packet number). When bash arrays approach 100,000 elements some things such as deleting entries slow down but that won't be an issue here. Other things that break down is yad to display the array in a dialog box but that shouldn't be an issue here either.
@DougSmythies yad could still be used to display the total bytes per IP address though.
@DougSmythies I have 4 more days off from work so if you'd like to paste your 10 minute data file some place (perhaps search and replace your own ip address with 999.9.9.9), I'll see if I can write a bash script without grep to analyze the raw data using arrays with a yad summary array of total bytes by IP address.
@WinEunuuchs2Unix : It is more complicated than that. My own work flow is considerably different than how I wrote my answer to your question. The file I was working with was a great many 10 minutes files all merged into 1, covering WAN data between 2017-05-31 08:09:33 and 2017-08-09 22:13:11, i.e. over two months. My outgoing file is 10266562589 bytes, and I processed the top 5000 IP addresses to check my top 100 assumption.
@WinEunuuchs2Unix : I tend to post-process tcpdump data in large chunks. Typically a month or more at a time.
@WinEunuuchs2Unix : Unless something comes up that needs more immediate attention.
@WinEunuuchs2Unix : My speed up idea is to stuff that file through sort using field 13 as the sort criteria (already done), and then modify that c program to parse bytes per IP address all in one pass through the file. I only know how in c, but there might be another way.
@DougSmythies Yes your 10 GB outgoing file is way too much for bash arrays to handle when only 8 GB of ram is available. I think bash limits array size to 4 GB anyway. That gets back to my original notion of using SQL to store packet data. I should seriously consider getting an ISP modem only solution like yourself and totally ditching the wifi router which I don't use anyway (too flakey in high density apartment building setting of downtown Edmonton).
@DougSmythies I think c is the fastest route. The only possible enhancement would be read block size of 4 MB chunks or something instead of byte by byte reads (if that is happening which I don't know).
@DougSmythies did some thinking and I like your approach. The changes I would make is a daily cron job to collect tcpdump, summarize the 1 to 2 GB outgoing by IP address, total bytes, time of first packet, time of last packet, and finally empty tcpdump for next day. Might have to lock tcpdump, copy it, delete and unlock it as start of cron daily job. Maybe even make it an hourly job? Then I would use yad to display the summary records loaded into a bash array.
@WinEunuuchs2Unix : yes, that was basically my original plan. However, I settled into something less and never got back to it. The reason I keep the raw binaries for a couple of months is because sometimes, even often, I find I want to go back to earlier data after some event. It has been incredibly useful.
@WinEunuuchs2Unix : One time my ISP accused me of doing bad stuff, but I was able to go back through the data and prove that the issue was when my nephew had his apple computer on my LAN. He has now been banned from my LAN.
@DougSmythies Very interesting! Well teenage boys will be teenage boys I guess. There are all kinds of hacking scripts floating around the dark web anyone can get access to. From my point of view it's my 1/4 Scottish side that hates the thought of paying for someone to upload 70 gb of data against my will off my laptop last month. I simply want to find out who, what, where, when, why and how. Your outgoing packet logging provides the data in raw format which is brilliant for many applications. IMO.
Well 20 hours/week of overtime at work because of the new system has ended now. Time to go the mall and spend some $$$. Cyaz Later :)
I was thinking that it might be advantageous to have a user with permissions higher than the root user.
You see, I would like to keep all of the activities and almost all existing root user privileges exactly as they are now.
However, I would like the ability to deny privileges root on an extre...
Try open 443 port for SSL
nmap xxx.xxx.xxx.xxx
Starting Nmap 6.40 ( http://nmap.org ) at 2017-09-03 23:56 EET
Nmap scan report for Ubuntu1404x64 (xxx.xxx.xxx.xxx)
Host is up (0.000018s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
80/tcp op...
I have some experience with linux having played with raspberry Pis a bit but I am still a relative noob...
I am setting up a pc running XUbuntu and so far it is going pretty well. I am having issues getting wifi working though.
Initially I tried with my existing Netgear A6200 but was told tha...
I installed Linux Mint 18 on a separate partition on my PC.
In order to post threads about any problem that I will encounter:
Can I do so in Ask Ubuntu?
Or in the Linux Mint forums?
I was thinking that it might be advantageous to have a user with permissions higher than the root user.
You see, I would like to keep all of the activities and almost all existing root user privileges exactly as they are now.
However, I would like the ability to deny privileges root on an extre...
Try open 443 port for SSL
nmap xxx.xxx.xxx.xxx
Starting Nmap 6.40 ( http://nmap.org ) at 2017-09-03 23:56 EET
Nmap scan report for Ubuntu1404x64 (xxx.xxx.xxx.xxx)
Host is up (0.000018s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
80/tcp op...
