我尝试安装了NICT的EXIST

2 年 ago
科, 颖
5 minutes

我安装了NICT网络安全研究室于2019年3月15日发布的EXIST，并记录下了这个过程。

首先

EXIST是什么在NICT（国家信息通信研究院）的NICTER分析团队开发的Web应用程序EXIST中，NICTER分析团队使用它来进行日常调查和分析工作。它可以自动汇集来自社区和安全供应商等提供的网络威胁信息。

EXIST是一个网络应用程序，它可以汇集各种网络威胁情报并进行跨源检索。它从各种信息源中通过订阅和API获取网络威胁情报，并将其集中存储在EXIST的数据库中。用户可以通过WebUI或者WebAPI，在EXIST上使用特定关键字进行跨源搜索网络威胁情报。（引自：NICTER博客）

png1

环境构成

- OS: CentOS 7.6  
- DB: MariaDB 10.3.13
- Python 3.6.7

安装请按照GitHub的步骤进行安装。
如果操作系统处于代理环境中，请事先在/etc/profile中添加代理设置。

操作系统的更新

# sudo yum update -y  
# sudo yum upgrade -y
# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

Git的安装

# yum install git -y

# プロキシの設定は必要に応じて実施
# git config --global http.proxy http://proxy.example.com:port
# git config --global https.proxy http://proxy.example.com:port

Python3的安装添加Yum存储库并安装Python3.6。

# yum install -y https://centos7.iuscommunity.org/ius-release.rpm
# yum install python36u python36u-libs python36u-devel python36u-pip -y
# yum groupinstall 'development tools' -y
# python3.6 --version
Python 3.6.7

pip的升级版本由于Python自带的pip版本较旧，因此您可以使用–upgrade选项将其更新为最新版本。如果处于代理环境下，您可以添加–proxy http://proxy.example.com:port选项。

# pip3.6 install --upgrade pip
# pip3.6 --version
pip 19.0.3 from /usr/lib/python3.6/site-packages/pip (python 3.6)

源代码的克隆

# cd /opt
# git clone https://github.com/nict-csl/exist.git

安装Python模块如果在代理环境下，可以在pip命令中添加–proxy http://proxy.example.com:port。

# cd /opt/exist
# pip install -r requirements.txt

安装MariaDB如果在代理服务器环境下，需要在 ~/.curlrc 文件中添加代理信息。

proxy-user=username:password
proxy=http://proxy.example.com:port

# curl -sS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash
[info] Repository file successfully written to /etc/yum.repos.d/mariadb.repo.
[info] Adding trusted package signing keys...
[info] Succeessfully added trusted package signing keys.

# yum install MariaDB-server MariaDB-client -y
# systemctl start mariadb
# systemctl enable mariadb

mysql_secure_installation: MySQL安全安装进行MariaDB的最小安全设置。

# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): //初期インストール時はパスワードがないため、そのままEnter押下
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: //新しいパスワードを入力
Re-enter new password: //再度、新しいパスワードを入力
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

重新启动MariaDB

# systemctl restart mariadb

MariaDB的配置.
使用mysql -u root -p命令登录MariaDB，并进行存在性检查的设置。

MariaDB [(none)]> CREATE DATABASE intelligence_db;
MariaDB [(none)]> CREATE USER 'exist'@'localhost' IDENTIFIED BY 'Passw0rd';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON intelligence_db.* TO 'exist'@'localhost';
MariaDB [(none)]> quit

Django配置复制模板文件并创建配置文件。

# cp -p /opt/exist/intelligence/settings.py.template /opt/exist/intelligence/settings.py

更新settings.py文件中的信息。
需要更新的内容如下：

    • ALLOWED_HOSTS : FQDNの情報

 

    DATABASES : #MariaDBの設定 で設定した情報
ALLOWED_HOSTS = [
     'localhost',
     'xxx.xxx.xxx.xxx', # インストールサーバーのIPアドレス etc.
]

...(中略)...

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'intelligence_db',
        'USER': 'exist',
        'PASSWORD': 'Passw0rd',
        'HOST': '',
        'PORT': '',
        'OPTIONS': {
            'charset': 'utf8mb4',
            'init_command': 'SET character_set_connection=utf8mb4;'
                            'SET collation_connection=utf8mb4_unicode_ci;'
                            "SET NAMES 'utf8mb4';"
                            "SET CHARACTER SET utf8mb4;"
        },
    }
}

Django的迁移

# cd /opt/exist
# python3.6 manage.py makemigrations exploit reputation threat threat_hunter twitter twitter_hunter
# python3.6 manage.py migrate

Redis服务器的安装

# yum install redis -y
# systemctl start redis
# systemctl enable redis

芹菜的设置

# Name of nodes to start
# here we have a single node
CELERYD_NODES="localhost"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="intelligence"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

芹菜的自动启动设置

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=root
Group=root
EnvironmentFile=/etc/sysconfig/celery
WorkingDirectory=/opt/exist
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

防火墙的配置

# firewall-cmd --zone=public --add-service=http --permanent
# firewall-cmd --zone=public --add-service=https --permanent
# firewall-cmd --zone=public --add-port=8000/tcp --permanent
# firewall-cmd --reload

打开EXIST

# mkdir -p /var/log/celery; chown root:root /var/log/celery
# mkdir -p /var/run/celery; chown root:root /var/run/celery
# systemctl start celery.service
# systemctl enable celery.service
# python3.6 /opt/exist/manage.py runserver 0.0.0.0:8000

在Web浏览器中访问 http://<EXIST服务器的IP地址>:8000，如果出现以下屏幕，则表示启动完成。

exist.png

相关链接

    • 独立行政法人情報通信研究機構 (NICT)

 

    • GitHub

 

    NICTER Blog – サイバー脅威情報集約システム EXIST

参考信息

    • CentOS7にPython3系をインストールする手順

 

    • NICTの公開したサイバー脅威情報を自動集約できるEXISTをつくってみた(インストール編)

 

    • curlコマンドにてproxy設定

 

    • MariaDBのインストール/初期設定 [CentOS7]

 

    サイバー脅威情報集約システム EXIST を構築してみた #exist

bannerAds