kertish-dos

Kertish-DOS

Build Status GitHub tag (latest SemVer) Docker Pulls Go Report Card

Table of Contents

Introduction

Kertish-dos is a simple and highly scalable distributed object storage to store and serve billions of files. It is developed to cover the expectation for mass file storage requirements in isolated networks. It does not have security implementation.

What is it for?

Kertish-dos is developed to cover the traditional file storage requirements in a scalable way. Software will use the same path-tree structure virtually to keep the files but will not stuck in hard-drive limits of the machine. It is possible to create a fail-safe, highly available fast and scalable file storage.

Where can it be used?

Systems where they need huge storage space requirements to keep all the files logically together. For example; common cloud storage for micro services, video storage for a streaming services, and the like…

How shouldn’t be used?

Kertish-dos does not have any security implementation. For this reason, it is best to use it in a publicly isolated network. Also it does not have file/folder permission integration, so user base limitations are also not available.

How is the best usage?

Kertish-dos is suitable to use as a back service of front services. It means, it is better not to allow users directly to the file manipulation.

Architecture

Kertish-dos has 3 vital parts for a working farm. Manager-Node, Head-Node, Data-Node.

Manager-Node and Data-Node must not accept direct request(s) coming from the outside of the Kertish-dos farm. For file/folder manipulation, only the access point should be Head-Node.

Here is a sample scenario;

  1. User wants to upload a file (demo.mov - 125Mb) to /Foo/Bar as demo.mov
  2. Uses the Head-Node to upload file with /Foo/Bar/demo.mov header and the file content as binary
  3. Head-Node accepts the whole file and asks to Manager-Node where and how to put this file in data-node cluster(s)
  4. Manager-Node creates the best possible chunk map for Head-Node and reserve the spaces for those chunks in data-node(s)
  5. Head-Node divides the file into chunks and push the chunks to related data-node(s) and at the end commit the provided map reservation if it has successful placement or discard the reservation to save the space.
  6. User gets success or error response with http status header.
  7. User can hold the file location in a database (/Foo/Bar/demo.mov)

Terminology

Summary

Kertish-dos allows you to create data farm in distributed locations. Every data farm consist of clusters. Clusters are data banks. Every cluster has one or more data node. Always one data node works as master. Every data node additions to that cluster will be used as backup of the master.

Data is always written to and deleted from the master data node in the cluster. However, reading request will be balanced between slave data nodes. So in read sensitive environments, as much as new data node added to the cluster will help you to increase the response time.

Features

System Requirements

Kertish-dos nodes has different hardware requirements to work flawless.

Setup Description

Kertish-dos farm consist of minimum

Head Node is for file storage interaction. When the data is wanted to access, the application should make the request to this node. It works as REST service. File storage command-line tool communicate directly to head node. Head node is scalable. Check head-node folder for details.

Manager Node is for orchestrating the cluster(s). When the system should be setup first time or manage farm for adding, removing cluster/node, this node will be used. Admin command-line tool communicate directly with manager node. Manager node is NOT scalable for now. Check manager-node folder for details.

Data Node is to keep the data blocks. All the file data particles will be distributed on data nodes in different clusters.

CAUTION: Deletion of a data node from cluster may cause the data lost and inconsistency.

Sample Setup

I’ll setup a farm using

Whole setup will be done on the same machine. This is just for testing purpose. In real world, you will need 6 servers for this setup. It is okay to keep the Mongo DB, Redis DSS, Locking-Center Server and Manager Node in the same machine if it covers the DB and DSS expectations.

You will not find the Mongo DB, Redis DSS, Locking-Center Server setup. Please follow the instruction on their web sites.

Setup Using Docker

The docker hub page is [https://hub.docker.com/r/freakmaxi/kertish-dos]

You can use the sample docker-compose file to kickstart the Kertish-dos farm in docker container with 6 Data-Nodes working in 3 Clusters as Master/Slave

[https://github.com/freakmaxi/kertish-dos/blob/master/docker-compose.yml]

docker-compose up will make everything ready for you.

Download setup script from [https://github.com/freakmaxi/kertish-dos/blob/master/kertish-docker-setup.sh]

Your Kertish-dos farm is ready to go.

Put any file using krtfs file storage tool. Ex:

./krtfs cp local:~/Downloads/demo.mov /demo.mov

Just change the path and file after local: according to the file in your system. Try to choose a file more than 70 Mb to see file chunk distribution between clusters. If file size is smaller than 32 Mb, it will be placed only in a cluster.

./krtfs ls -l will give you an output similar like below

processing... ok.
total 1
- 87701kb 2020 Jun 22 22:07 demo.mov

Setup Using Binaries

Preparation

Setting Up Manager Node

You can take a look at Manager-Node page to understand how it is working

export MONGO_CONN=”mongodb://root:pass@127.0.0.1:27017” # Modify the values according to your setup export REDIS_CONN=”127.0.0.1:6379” # Modify the values according to your setup export LOCKING_CENTER=”127.0.0.1:22119” # Modify the values according to your setup /usr/local/bin/kertish-manager

- Give execution permission to the file `sudo chmod +x [Saved File Location]`
- Execute the saved file.
---
##### Setting Up Head Node

You can take a look at [Head-Node](https://github.com/freakmaxi/kertish-dos/blob/master/head-node) page to understand how it is working

- Copy `kertish-head` executable to `/usr/local/bin` folder on the system.
- Give execution permission to the file `sudo chmod +x /usr/local/bin/kertish-head`
- Create an empty file in your user path, copy-paste the following and save the file
```shell script
#!/bin/sh

export MANAGER_ADDRESS="http://127.0.0.1:9400" 
export MONGO_CONN="mongodb://root:pass@127.0.0.1:27017" # Modify the values according to your setup
export LOCKING_CENTER="127.0.0.1:22119"                 # Modify the values according to your setup
/usr/local/bin/kertish-head

You can take a look at Data-Node page to understand how it is working

export BIND_ADDRESS=”127.0.0.1:9430” export MANAGER_ADDRESS=”http://127.0.0.1:9400” export SIZE=”1073741824” # 1gb export ROOT_PATH=”/opt/c1n1” /usr/local/bin/kertish-data

- Give execution permission to the file `sudo chmod +x [Saved File Location]`
- Execute the saved file.

---

- Create an empty file on your user path named `dn-c1n2.sh`, copy-paste the following and save the file
```shell script
#!/bin/sh

export BIND_ADDRESS="127.0.0.1:9431"
export MANAGER_ADDRESS="http://127.0.0.1:9400"
export SIZE="1073741824" # 1gb
export ROOT_PATH="/opt/c1n2"
/usr/local/bin/kertish-data

export BIND_ADDRESS=”127.0.0.1:9432” export MANAGER_ADDRESS=”http://127.0.0.1:9400” export SIZE=”1073741824” # 1gb export ROOT_PATH=”/opt/c2n1” /usr/local/bin/kertish-data

- Give execution permission to the file `sudo chmod +x [Saved File Location]`
- Execute the saved file.

---

- Create an empty file on your user path named `dn-c2n2.sh`, copy-paste the following and save the file
```shell script
#!/bin/sh

export BIND_ADDRESS="127.0.0.1:9433"
export MANAGER_ADDRESS="http://127.0.0.1:9400"
export SIZE="1073741824" # 1gb
export ROOT_PATH="/opt/c2n2"
/usr/local/bin/kertish-data

IMPORTANT: Data nodes sizes in the SAME CLUSTER have to be the same. You may have different servers with different sized hard-drives. You should use the SIZE environment variable to align the storage spaces according to the the server that has the smallest hard-drive size

ok.

- Enter the following command to create the one another cluster
`krtadm -create-cluster 127.0.0.1:9432,127.0.0.1:9433`
- If everything went right, you should see something like this

Cluster Details: 8f0e2bc02811f346d6cbb542c92d118d Data Node: 127.0.0.1:9432 (MASTER) -> 7a758a149e4453b20a40b35f83f3a0e4 Data Node: 127.0.0.1:9433 (SLAVE) -> 6776201a0bb7daafb46c9e3931f0807e

ok.

---
##### Manipulating File Storage

- Copy `krtfs` executable to `/usr/local/bin` folder on the system.
- Give execution permission to the file `sudo chmod +x /usr/local/bin/krtfs`
- Enter the following command
`krtfs ls -l`
output:

processing… ok. total 0

- Put a file from your local drive to dos
`krtfs cp local:/usr/local/bin/krtfs /krtfs`
output:

processing… ok.

- Enter the following command
`krtfs ls -l`
output:

processing… ok. total 1

If you get the same or similar outputs like here, congratulations! you successfully set up your Kertish-dos.

One Last Important Note

When you setup the cluster and the cluster starts taking data blocks, consider that cluster is as absolute. Deleting the cluster will cause you data inconsistency and lost. Due to this reason, when you are creating the structure of your farm, pay attention to your cluster setups the most. If you want to remove the cluster from the farm, consider to move the cluster first from one point to another using krtadm client tool.

This site is open source. Improve this page.