Catch the fish - How to check for unwanted network traffic on solaris

During a calm and rainy day at work, I suddenly get a security request to explain why a certain server is making http calls to the outside. As the environment is pretty complex and many teams are hosting their applications, it is somehow fustrating to not find back the answer within 5 minutes.
All documentation available in regards the developed apps, were not mentioning the networkflow which was questioned. Nor did the command "netstat -an" show any result.

So I did a simple check of each PID to see if the command "pfiles PID" would reveil the wanted traffic. But again nothing to find. The process must have triggered very fast and died very fast before I could catch this fish.

Time to start fishing. A small script, using the command "lsof -i4" would run in a loop, constantly checking if my traffic is present. And if it does, then ptree will capture the process info.

IFS=" " rm /var/tmp/lsof.tmp 2>/dev/null; touch /var/tmp/lsof.tmp while [ "`cat /var/tmp/lsof.tmp | grep "$1"`" = "" ]; do   lsof -i4 > /var/tmp/lsof.tmp 2>/dev/null done echo "TARGET SPOTTED ON: "`date "+%Y%m%d %H:%M:%S"` echo "-------------------------------------------------------------------" for line in `cat /var/tmp/lsof.tmp | grep "$1"`; do   echo $line   ptree `echo $line | awk '{print $2}'`   echo "-------------------------------------------------------------------" done echo "FINISHED" rm /var/tmp/lsof.tmp

And guess what happened, I caught the fish:

Hercules-root# ./valid_network_check.ksh 134.20 TARGET SPOTTED ON: 20160601 22:09:47 ------------------------------------------------------------------- java       1144   tomcat   12u  IPv4 0x3026601ea40         0t0  TCP server1:62153->134.20.87.65:80 (SYN_SENT) 1144  /var/as/java/jre1.7.0_67/bin/java -Dsdc.tc.id=as01_ti01 -Djava.util.l ------------------------------------------------------------------- java       1144   tomcat   14u  IPv4 0x30175cfa7c0         0t0  TCP server1:62227->134.20.87.65:80 (SYN_SENT) 1144  /var/as/java/jre1.7.0_67/bin/java -Dsdc.tc.id=as01_ti01 -Djava.util.l ------------------------------------------------------------------- java      25741   tomcat  147u  IPv4 0x304b1cc4340         0t0  TCP server2:62172->134.20.88.65:80 (SYN_SENT) 25741 /var/as/java/jre1.8.0_60/bin/java -Dsdc.tc.id=as01_ti01 -Djava.util.l ------------------------------------------------------------------- java      25741   tomcat  148u  IPv4 0x301a31fd580         0t0  TCP server2:62223->134.20.88.65:80 (SYN_SENT) 25741 /var/as/java/jre1.8.0_60/bin/java -Dsdc.tc.id=as01_ti01 -Djava.util.l ------------------------------------------------------------------- FINISHED

 This info was for me enough to escalate to the development team, so that they could take the action needed.

Time again to fetch some coffee and to start a new quest. :-)