Mentioning tcpflow here b/c it's one of the most useful networking related tools I know of that very few people even know about.
What does it do?
It can reassemble TCP packets back into the FULL body of the original message sent. e.g. if you make a HTTP GET request, it will show you the full text in a file stamped with the time, source and dest ips and port.
Things I've found it REALLY useful for:
- migrating a data center
- for some reason, connection works fine on the old DC but seems to time out in weird ways in the new DC
- No one can figure it out
- I suggest using tcpflow
- Turn out there was a setting in the new DC network hardware that was truncating larger packets and the authorization message was just over the threshold
People always say "yeah, but Wireshark" which is true, that's a good tool too. That being said, there is just something about seeing the "raw" text of a message sent by a machine over the wire and being able to see it in text from the command line.
`tshark` (from wireshark) sounds like a similar tool with pretty good protocol support (IIRC, it has been a long time since I've used it).
By example:
$ tshark -i lo -l -q -Y http -T fields -e frame.time -e _ws.col.Info -e http.file_data
Running as user "root" and group "root". This could be dangerous.
Capturing on 'Loopback: lo'
Dec 19, 2022 04:07:12.419196072 UTC GET / HTTP/1.0
Dec 19, 2022 04:07:12.420560079 UTC HTTP/1.1 200 OK (text/plain) Hello World!\n
Dec 19, 2022 04:07:12.504444583 UTC GET /favicon.ico HTTP/1.0
Dec 19, 2022 04:07:12.505691738 UTC HTTP/1.1 404 Not Found
Because -l (flush every line to stdout) and -q (suppress packet counts) you can pipe the command to grep.
$ tshark -i lo -l -q -Y http -T fields -e frame.time -e _ws.col.Info -e http.file_data 2>&1 | grep 404
Dec 19, 2022 04:09:46.361304160 UTC HTTP/1.1 404 Not Found
I've been using tcpick for a similar purpose to see if a HTTP request is reaching and not blocked elsewhere and to see the raw communication to check on headers.
Thanks, this'll be interesting. One of my first go-tos in Wireshark is finding the interesting hosts I'm looking for, then using the built-in Follow TCP Stream to go deeper.
I wonder if tcpflow would make this easier, or maybe just different, by effectively pre-slicing the capture into sessions.
> The Nmap Scripting Engine (NSE) allows you to use, and share various scripts. The scripts are written in Lua.
Stuff like this is why I can't take people who argue against systemd because of the Unix philosophy seriously.
The idea of this mythical program which does exactly one thing is just that, mythical. Yeah, modularity is important, but these programs are also made for real people who'd rather just use a command flag than figure out how to chain 50 different programs together in a brittle Rube Goldberg machine to get the result they want. Just look at how many command flags GNU ls has!
The Unix philosophy is a good one, but I'd rather judge program's based on their actual usability.
Without weighing into systemd debates on which I am not qualified, there is a difference between including the kitchen sink in an app that you don't have to use or have installed at all, eg emacs can be eschewed in favour of nano, ed, nvi or whatever vs a core part of the system you are using.
The unix philosophy is merely an explanation of a preference in building things. It can be ignored by yourself where appropriate, be a minor concern or be considered the key guiding and crucial principle in your decision making.
Whatever you think of systemd its criteria for whether you think it good, bad or indifferent design in principle is likely to be pretty different to emacs.
Rob Pike once said the unix philosophy is dead with Perl reading the Euology. (...and didn't Plan 9 exploit that to world dominance? YMMV)
May everyone enjoy using the systems that they use or at least be getting closer to that ideal.
I'm a former nmap core developer and even though I no longer actively contribute the project still is very close to my heart. It wasn't "revolutionary" at first, in that port scanning was already a thing, there were multiple available scanners out there. Most of these tools had their own syntax though, and they usually offered one specific type of scan. So nmap came and aggregated everything into a relatively simple tool. It then became incredibly successful and from that popularity nmap got its greatest strengths IMHO : portability, versatility (these crazy options implemented to satisfy super-specific needs) and the OS and service version signatures captured as a collaborative effort over more than a decade!
Nmap's reference guide is very good too. Simple yet thorough. Learned a lot about networking just by reading through this for some example on host discovery. https://nmap.org/book/man.html
I really wish nmap has a proper API, or programmatically-friendly pipeline output.
All of its myriad output formats are just "skins" on top of the pretty-printed textual UI output, making nmap very difficult to use as a component of a larger system.
My first impression of this tool when I first used it a couple of years back was "Oh okay, a tiny little tool to find the IP address of a device you want to ssh on". Only over time I realised how powerful it actually is.
What does it do?
It can reassemble TCP packets back into the FULL body of the original message sent. e.g. if you make a HTTP GET request, it will show you the full text in a file stamped with the time, source and dest ips and port.
Things I've found it REALLY useful for:
- migrating a data center
- for some reason, connection works fine on the old DC but seems to time out in weird ways in the new DC
- No one can figure it out
- I suggest using tcpflow
- Turn out there was a setting in the new DC network hardware that was truncating larger packets and the authorization message was just over the threshold
People always say "yeah, but Wireshark" which is true, that's a good tool too. That being said, there is just something about seeing the "raw" text of a message sent by a machine over the wire and being able to see it in text from the command line.
https://github.com/simsong/tcpflow