Convert curl to python request

While writing some code, I stumbled across a API documentation, that only had curl examples (prefer to have curl examples over no examples at all) but I had some troubles converting it to proper python code and a friend recommended a page called: https://curl.trillworks.com/

Convert curl syntax to Python, Node.js, R, PHP, Strest, Go, JSON, Rust

And it is even available on github. How cool is that?

This blogpost is only to save it as kind of a bookmark for future coding adventures.

Combining Virustotal, PassiveSSL and Timesketch

Motivation

Playing with Timesketch for a while and working on some OSINT timelines I was tired to investigate MD5 and domains / ips all manually so I tried to automate some of the work. Why is that important? If you have a list of hashes, domains and IPs, you of course can check your SIEM, EDR solution etc – but what if you have a hit? Would it benefit your investigation to at least have an idea of the timeframe something was used by attackers or seen in the wild?

Most shared indicators are lacking the timeframe, so we need to add those values by external information on our own.

Virustotal

There is no need to further explain Virustotal, it is basically a huge dataset of malware and information about domains and ips.

In particular information about a specific point in time a domain was seen to point to an IP and back is good to know to build your timeline.

E.g. if you have verymalicious.com pointing to 127.0.0.1 all the time, only on one day it was pointing to 1.2.3.4 – hits in your infrastructure should be higher escalated if seen during that day, out of that time window it might still be important, but not as urgent as during that day.

In regards to hash intelligence, Virustotal is nice, because if you add the info, when the last scan date of a file was, you can at least tell, that the file was known after that day.

I asked Virustotal to add more information they already have to the API and we will have wait till it is exposed:

  • First seen in the wild
  • First uploaded to VT
  • PE compile time

PassiveSSL

Alexandre Dulaunoy and Eireann Leverett have given a talk at the FIRST conference in Berlin back in 2015, which took my attention, but it took some time till I really had time to implement something to use the idea.

The basic idea is that, out of several sources, passive ssl services such as CIRCL passiveSSL collect certificates and expose information via API.

For timeline analysis in particular, the following dates are important as they might shine some light of attacker activity:

  • first seen in the wild
  • last seen in the wild
  • not use before
  • not use after

If you now add all of the information above, you might be able to get a better idea, when an IP / Domain / File was active.

This information should then be fed into a Timesketch investigation.

Example

Using some sample data from APT33: https://github.com/deralexxx/osint-timelines/blob/master/2018/2018-12-21.OVERRULED:-Containing-a-Potentially-Destructive-Adversary-APT33.csv

Combining with the python script below with the following indicators:

basepack.org
103.236.149.124
5.79.66.241
8d3fe1973183e1d3b0dbec31be8ee9dd
fa7790abe9ee40556fb3c5524388de0b

Findings

Domains resolving to IPs

It is transparent when the hashes have been last scanned and what Ips resolve to the domains mentioned in the report.

The other thing is that right before some malware was mentioned by Fireeye in the report, SSL certificates became invalid:

SSL Certificate

Of course the individual SSL certificate can also been investigated:

Python

The example is available on github: https://github.com/deralexxx/osint_to_timesketch

Next steps

  • Waiting for VT to expose more things
  • Improve the script
  • Introduce multiple pDNS providers

timesketch-tools

Overview

I am happy to say that a new tool made it to github called „timesketch-tools“.
It is basically a way to interact with Timesketch via CLI. For those who don’t know Timesketch, it is an amazing opensource tool developed by Johan Berggren and is used to create timelines for forensic investigations as well as incident response cases.

Reason

Back in 2017, Johan tweeted:

Why is the WebUi not enough? Well in some cases you might want to automate stuff, have no browser or other reasons, so it is not „Why“ but „why not“.

So I did during the last few days and built a client for it: timesketch-tools

Capabilities

At the moment only two methods do work, but it should be enough to show the power of it.

List sketches

timesketch-tools.py -ls
     
         _______               __       __      __ 
        /_  __(_)_ _  ___ ___ / /_____ / /_____/ / 
         / / / /  ' \/ -_|_-</  '_/ -_) __/ __/ _          
        /_/ /_/_/_/_/\__/___/_/\_\__/\__/\__/_//_/-tools v0.1

            
+-----+-----------------------------+
|  id |             Name            |
+-----+-----------------------------+
| 130 |     test1Untitled sketch    |
|  3  | The Greendale investigation |
+-----+-----------------------------+

Add event

timesketch-tools.py --add_events
     
         _______               __       __      __ 
        /_  __(_)_ _  ___ ___ / /_____ / /_____/ / 
         / / / /  ' \/ -_|_-</  '_/ -_) __/ __/ _          
        /_/ /_/_/_/_/\__/___/_/\_\__/\__/\__/_//_/-tools v0.1

            
Please provide the sketch id you want to add events to as (an integer): 3
Please provide informations to the event you would like to add timestamp, timestamp_desc, message will be promted

Timestamp (use Format: YYYY-mm-ddTHH:MM:SS+00:00 2018-01-15T10:45:50+00:00) use c for current time c
timestamp_desc this is the description
message something was hacked
Event added, ID: 18 Date:2018-10-31T14:49:41+00:00 timestamp desc this is the description messagesomething was hacked
Add another event? (y/n)n

I have a lot of ideas to improve, so expect some more functionality added soon…

curl -u in python

Problem

Sometimes you might want to authenticate against an API with username and password where examples are only listed with curl:


curl -u username:password https://127.0.0.1/foobar

Solution

If you want to implement the same in python you can use the following


import requests
from requests.auth import HTTPBasicAuth
username = "username"
password = "password"

request_url = "https://127.0.0.1/foobar"

result = requests.post(request_url, auth=HTTPBasicAuth(username, password))

Hope it helps, let me know

Preis: EUR 48,99
statt: EUR 49,99

fatal error: ‚fuzzy.h‘ file not found

Trying to install pydeep on OSX and getting the error below:

pydeep.c:2:10: fatal error: 'fuzzy.h' file not found
#include
^~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

So far I have not found a solution for it.

Some people recommended to use:

xcode-select --install

Which did not help my issues.

Also the following:

$ export LDFLAGS="-L/usr/local/lib"
$ export C_INCLUDE_PATH=/usr/local/include

Did not help

Still searching for a solution and will update that blogpost once I found anything

Raspberry Pi EyeFi Server

I tried to ceate a Raspberry Pi as a standalone Photo catching device for multile EyeFi Cards.

Turns out that is not possible at the moment using EyeFi Mobi cards.

That is what I tried:

Hardware:

– Raspberry Pi

– EyeFi Mobi card

– Edimax USB Wifi Dongle

– Camera

Software:

– Raspian install
– https://github.com/dgrant/eyefiserver2/

Installation:

– git clone the eyefiserver2
– follow https://github.com/dgrant/eyefiserver2/wiki/Getting-Started

Starting

Start the script

sudo eyefiserver.py start /etc/eyefiserver.conf /var/log/eyefiserver.log

Check

[03/26/16 01:32PM][runEyeFi] - Eye-Fi server started listening on port 59278
tcp        0      0 0.0.0.0:59278           0.0.0.0:*               LISTEN      873/python  

Seems okay

Upload Key

The first issue was the upload key.
Connected two different eyefi cards with OSX and Windows 7 and was unable to find an upload key other then 00000000000000000000000000000000

On OSX:

/Users/$USERNAME/Library/Application Support/Eyefi/Eyefi Mobi/

But there is a SQL database in:

And you can do the following:

sqlite3 offline.db
SQLite version 
Enter ".help" for usage hints.
sqlite> SELECT o_mac_address, o_upload_key FROM o_devices;
00-11-11-11-11-11|12345678901234567890123456789012

Hm but still, using that upload key (was reducted) the eyefiserver2 did not work.

And I was unable to get a connection from my camera to my pi.

There is an issue reported in github:

https://github.com/dgrant/eyefiserver2/issues/9

That referenced the following Whitepaper:

https://www.os3.nl/_media/2013-2014/courses/ot/connor_stavros.pdf

So at the moment the problem has not been solved,an workaround would be using an Mac / Windows System, or to upgrade to the larger EyeFi Version:

Feel free to comment your solutions below.

Further reading:

Raspberry PI and Eye-Fi
http://support.photosmithapp.com/knowledgebase/articles/116903-why-do-i-see-multiple-eye-fi-card-upload-keys-ho
https://github.com/michaelbrandt/node-eyefimobiserver/blob/master/related_work/eyefi-mobi.py
http://www.ephototag.com/using-a-eye-fi-card/
https://launchpad.net/eyefi/+download
http://bazaar.launchpad.net/~jordens/eyefi/trunk/view/head:/README.rst
http://support.photosmithapp.com/knowledgebase/articles/152395-how-do-i-generate-an-eye-fi-card-upload-key
http://thewifibooth.com/article/eyefiuploadkey-x2pro/

Adding your own crt from a CA to ubuntu local ca store

If you are trying to for example develop python and accessing something encrypted with SSl and that SSL certificate is not signed by a well known CA you might get an error.

That is because your CA is not added to the local CA store of e.g. Ubuntu.

You can add your signatures by:

sudo mkdir /usr/share/ca-certificates/extra
sudo cp FOO.crt /usr/share/ca-certificates/extra/FOO.crt
sudo dpkg-reconfigure ca-certificates

Then the new certificates will be added to your local store.
(please be careful as sudo dpkg-reconfigure ca-certificates is only checking for files *.crt, so no *.cer etc).

Python PyDev Eclipse Ubuntu

To Use PyDev Eclipse Plugin on Ubuntu:


apt-get install eclipse
open eclipse
help --> Install New Software
add button
insert: PyDev
Position: http://pydev.org/updates
check all
accept license
Next
Window -> Preferences -> PyDev -> Interpreter -> Python
New -> link to python interpreter (default: /usr/bin/python)
finish