DLib를 사용하여 Android에서 얼굴 검출(Face Landmarks Detection)하는 예제OpenCV/Dlib2023. 10. 13. 22:33@webnautes

Table of Contents

DLib를 사용하여 Android에서 얼굴 검출(Face Landmarks Detection)하는 방법을 진행하여 보았습니다.

캡처 버튼을 클릭하여 사진을 찍고, 검출 버튼을 눌러서 얼굴을 검출합니다.

인터넷에서 찾아본 결과와 달리 검출 시간이 오래 걸리는 편입니다.

속도를 최적화하는 방법은 찾아보면 있을듯합니다.

이미지를 작게하면 빨라질 가능성도 있습니다.

본 포스트는 다음처럼 구성으로 되어있습니다.

1. DLib 라이브러리 컴파일

2. 안드로이드 프로젝트 생성

3. 얼굴 검출(Face Landmarks Detection) 예제

마지막 업데이트

2018. 8. 24

2019. 5. 27 dlib 빌드 방법 수정

2019. 7. 16 진행에 혼동을 줄 수 있는 build.gradle 복사본 수정

2020. 7. 23 최신 버전 Android SDK, NDK에 맞추어 동작하도록 수정

1. DLib 라이브러리 컴파일

진행하기 전에 SDK Manager에서 CMake와 NDK를 설치하세요.

아래쪽에 보이는 Show Package Details를 체크하고 21.3.6528147 버전을 설치했는지 확인해보세요.

글 작성 시점에서 최신 버전입니다.

글작성 시점에서 SDK Platform 최신버전인 Android API 30을 설치하여 사용합니다.

1-1. 깃허브에서 Dlib 소스코드를 다운로드합니다.

https://github.com/davisking/dlib/releases

1-2. 압축을 풀어 작업 편의상 사용자 디렉토리( C:\Users\사용자)로 복사해줍니다.

압축 푸는 방법에 따라 dlib-19.20 폴더 안에 또 dlib-19.20 폴더가 있을 수 있으니 꼭 스크린샷처럼 보이는지 확인하세요.

1-3. cmake 명령을 하기 전에 옵션으로 사용하는 디렉토리 위치를 확인합니다.

아래 경로에 있는 숫자로 된 폴더 중 하나를 선택합니다. 숫자 높은 것이 최신버전의 NDK입니다.

C:\Users\사용자이름\AppData\Local\Android\Sdk\ndk\

여기에선 21.3.6528147 폴더를 선택하여 다음 두 옵션으로 사용합니다. 숫자 부분빼고는 동일한 경로입니다.

ANDROID_NDK로 지정한 경로의 하위 디렉토리에 CMAKE_TOOLCHAIN_FILE에서 지정한 파일이 존재합니다.

-DCMAKE_TOOLCHAIN_FILE=C:\Users\webnautes\AppData\Local\Android\Sdk\ndk\21.3.6528147\build\cmake\android.toolchain.cmake

-DANDROID_NDK=C:\Users\webnautes\AppData\Local\Android\Sdk\ndk\21.3.6528147

윈도우 키 + R을 누른후, cmd 엔터를 입력하여 명령 프롬프트 창을 실행합니다.

소스 디렉토리로 이동 후, cmake를 실행합니다.

파란색 부분은 현재 윈도우에 로그인한 사용자 이름으로 변경하세요.
빨간색 부분은 빌드하여 사용할 플랫폼 이름으로 변경하세요.
초록색 부분은 앞에서 살펴본 NDK 버전으로 일치해야 합니다.

C:\Users\webnautes>cd dlib-19.20

C:\Users\webnautes\dlib-19.20>mkdir build

C:\Users\webnautes\dlib-19.20>cd build

ANDROID_PLATFORM으로 android-30을 사용합니다.

C:\Users\webnautes\dlib-19.20\build>C:\Users\webnautes\AppData\Local\Android\Sdk\cmake\3.10.2.4988404\bin\cmake -DBUILD_SHARED_LIBS=1 -DCMAKE_TOOLCHAIN_FILE=C:\Users\webnautes\AppData\Local\Android\Sdk\ndk\21.3.6528147\build\cmake\android.toolchain.cmake -DANDROID_NDK=C:\Users\webnautes\AppData\Local\Android\Sdk\ndk\21.3.6528147 -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=ninja -DCMAKE_CXX_FLAGS=-std=c++11 -frtti -fexceptions -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-30 -DDLIB_NO_GUI_SUPPORT=OFF -DANDROID_TOOLCHAIN=clang -DANDROID_STL=c++_shared -DANDROID_CPP_FEATURES=rtti exceptions -DCMAKE_INSTALL_PREFIX=C:\dlib-native ..

문제 없으면 마지막에 다음처럼 출력됩니다.

-- Configuring done

-- Generating done

-- Build files have been written to: C:/Users/webnautes/dlib-19.20/build

1-4. 빌드를 진행합니다.

-j 뒤에는 사용하는 CPU의 코어 갯수를 입력하세요. 빌드 속도가 빨라집니다.
파란색 부분은 현재 윈도우에 로그인한 사용자 이름으로 변경하세요.

C:\Users\webnautes\dlib-19.20\build>C:\Users\webnautes\AppData\Local\Android\Sdk\cmake\3.10.2.4988404\bin\ninja.exe -j4

문제 없으면 마지막에 다음처럼 출력됩니다.

1-5. 결과물을 설치합니다. 앞에서 cmake 실행시 옵션으로 지정한 C:\dlib-native에 라이브러리 파일과 헤더파일을 복사해줍니다.

파란색 부분은 현재 윈도우에 로그인한 사용자 이름으로 변경하세요.

C:\Users\webnautes\dlib-19.20\build>C:\Users\webnautes\AppData\Local\Android\Sdk\cmake\3.10.2.4988404\bin\ninja.exe install

1-6. 빌드 결과물이 다음 경로에 저장됩니다.

2. 안드로이드 프로젝트 생성

2-1. 프로젝트 생성시 Native C++을 선택합니다.

2-2. Name 항목에 프로젝트 이름을 적어주고 Language를 Java로 합니다.

3. 얼굴 검출(Face Landmarks Detection) 예제

4-0. 프로젝트가 생성되기를 기다립니다. 안드로이드 스튜디오 하단에 다음 메시지가 사라져야 합니다.

프로젝트 창을 Project 뷰로 변경합니다.

4-1. 프로젝트 창에서 app / src / main을 선택한 상태에서 마우스 우클릭하여 보이는 메뉴에서 New > Directory를 선택합니다.

assets 라고 입력하고 OK 버튼을 누르면 assets 이름의 디렉토리가 생성됩니다.

아래 링크에서 Download 버튼을 클릭하여 shape_predictor_68_face_landmarks.dat 파일을 다운로드합니다.

https://github.com/AKSHAYUBHAT/TensorFace/blob/master/openface/models/dlib/shape_predictor_68_face_landmarks.dat

다운로드 받은 파일을 선택하고 Ctrl 키를 누른 상태에서 드래그해서 프로젝트 창의 assets 폴더에 놓으면 복사가 됩니다.

4-2. 미리 빌드해놓은 dlib를 안드로이드 프로젝트로 복사해줍니다.

Ctrl 키를 누른 채 dlib-native 폴더를 드래그하여 프로젝트 이름에 놓으면 복사가 됩니다.

복사후 다음 위치에 있어야 합니다.

4-3.activity_main.xml 파일의 내용을 다음 코드로 대체합니다.

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:id="@+id/layout_main"
android:layout_width="match_parent"
android:layout_height="match_parent" >

<SurfaceView
android:id="@+id/camera_preview_main"
android:layout_width="match_parent"
android:layout_height="match_parent"/>

<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:gravity="bottom"
android:orientation="vertical">

<LinearLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:gravity="center_horizontal"
android:orientation="horizontal">

<ProgressBar
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:id="@+id/progress1" />
</LinearLayout>

<LinearLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:gravity="center_horizontal"
android:orientation="horizontal">
<Button
android:id="@+id/button_main_capture"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="center_horizontal"
android:text="캡처" />

<Button
android:id="@+id/button_main_detection"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="center_horizontal"
android:text="검출" />
</LinearLayout>
</LinearLayout>
</FrameLayout>

4-4. layout 폴더에 dialog.xml을 추가합니다.

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:orientation="vertical"
android:gravity="center"
android:layout_width="wrap_content"
android:layout_height="wrap_content">

<ImageView
android:id="@+id/dialogImage"
android:layout_width="wrap_content"
android:layout_height="350dp"/>

</LinearLayout>

4-5. styles.xml에 노란색 코드를 추가합니다.

<resources>


<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">

<item name="colorPrimary">@color/colorPrimary</item>
<item name="colorPrimaryDark">@color/colorPrimaryDark</item>
<item name="colorAccent">@color/colorAccent</item>

<item name="windowActionBar">false</item>
<item name="windowNoTitle">true</item>
</style>

</resources>

4-6. build.gradle에 com.google.android.material을 추가합니다.

dependencies {
    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation 'androidx.appcompat:appcompat:1.1.0'
    implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    implementation 'com.google.android.material:material:1.3.0-alpha01'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
}

build.gradle에 다음 옵션을 추가해주고 Sync Now를 눌러 적용해줍니다.

기존에 있는 externalNativeBuild을 지우고 아래 노란색 줄로 바꿔주면됩니다. abiFilters에는 사용할 플랫폼만 적어줍니다.

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.0"

    defaultConfig {
        applicationId "com.tistory.webnautes.dlibexample"
        minSdkVersion 21
        targetSdkVersion 30
        versionCode 1
        versionName "1.0"

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                arguments "-DANDROID_STL=c++_shared" , "-DANDROID_TOOLCHAIN=clang"
                cppFlags "-std=c++11 -fexceptions"
            }
        }
        ndk {
            abiFilters "arm64-v8a"
        }

    }

4-7. MainActivity.java의 내용을 다음 코드로 변경합니다.

/*
* https://github.com/josnidhin/Android-Camera-Example에 있는 코드를 수정했습니다.
*/

package com.tistory.webnautes.dlibexample;

import androidx.appcompat.app.AppCompatActivity;
import android.os.Bundle;
import android.Manifest;
import android.content.DialogInterface;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.hardware.Camera;
import android.net.Uri;
import android.os.AsyncTask;
import android.os.Environment;
import androidx.annotation.NonNull;
import com.google.android.material.snackbar.Snackbar;
import androidx.core.app.ActivityCompat;
import androidx.core.content.ContextCompat;
import androidx.appcompat.app.AlertDialog;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.SurfaceView;
import android.view.View;
import android.view.Window;
import android.view.WindowManager;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.ProgressBar;

import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;

public class MainActivity extends AppCompatActivity
implements ActivityCompat.OnRequestPermissionsResultCallback {

private static final String TAG = "android_camera_example";
private static final int PERMISSIONS_REQUEST_CODE = 100;
String[] REQUIRED_PERMISSIONS = {Manifest.permission.CAMERA,
Manifest.permission.WRITE_EXTERNAL_STORAGE};
private static final int CAMERA_FACING = Camera.CameraInfo.CAMERA_FACING_FRONT;//Camera.CameraInfo.CAMERA_FACING_BACK; //

private SurfaceView surfaceView;
private CameraPreview mCameraPreview;
private View mLayout; // Snackbar 사용하기 위해서는 View가 필요합니다.
// (참고로 Toast에서는 Context가 필요했습니다.)
private ProgressBar mProgressBar;

public native void Detect();

// Used to load the 'native-lib' library on application startup.
static {
System.loadLibrary("dlib");
System.loadLibrary("native-lib");
}

private void copyFile(String filename) {
String baseDir = Environment.getExternalStorageDirectory().getPath();
String pathDir = baseDir + File.separator + filename;

AssetManager assetManager = this.getAssets();

InputStream inputStream = null;
OutputStream outputStream = null;

try {
Log.d( TAG, "copyFile :: 다음 경로로 파일복사 "+ pathDir);
inputStream = assetManager.open(filename);
outputStream = new FileOutputStream(pathDir);

byte[] buffer = new byte[1024];
int read;
while ((read = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, read);
}
inputStream.close();
inputStream = null;
outputStream.flush();
outputStream.close();
outputStream = null;
} catch (Exception e) {
Log.d(TAG, "copyFile :: 파일 복사 중 예외 발생 "+e.toString() );
}

}

private class Task extends AsyncTask<Void, Void, Void> {

@Override
protected void onPreExecute() {
super.onPreExecute();
mProgressBar.setVisibility(View.VISIBLE);
}

@Override
protected void onPostExecute(Void aVoid) {
super.onPostExecute(aVoid);

Log.d("MainActivity", "end");
mProgressBar.setVisibility(View.GONE);

// 갤러리에 반영
File a = new File("/storage/emulated/0/camtest/output.bmp");
Intent mediaScanIntent = new Intent( Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
mediaScanIntent.setData(Uri.fromFile(a));
sendBroadcast(mediaScanIntent);

// stackoverflow.com/a/28128256
AlertDialog.Builder builder = new AlertDialog.Builder(MainActivity.this);
builder.setPositiveButton("Close", new DialogInterface.OnClickListener() {
@Override
public void onClick(DialogInterface dialog, int which) {
}
});
final AlertDialog dialog = builder.create();
LayoutInflater inflater = getLayoutInflater();
View dialogLayout = inflater.inflate(R.layout.dialog, null);
dialog.setView(dialogLayout);
dialog.requestWindowFeature(Window.FEATURE_NO_TITLE);

dialog.setOnShowListener(new DialogInterface.OnShowListener() {
@Override
public void onShow(DialogInterface d) {
ImageView imageView = (ImageView) dialog.findViewById(R.id.dialogImage);

File imgFile = new File("/storage/emulated/0/camtest/output.bmp");
Bitmap bitmap = BitmapFactory.decodeFile(imgFile.getAbsolutePath());
int height = bitmap.getHeight();
int width = bitmap.getWidth();

Bitmap resized = null;

float dpi = getResources().getDisplayMetrics().density;
int px = (int)(350 * dpi);

if(width > px) {
resized = Bitmap.createScaledBitmap(bitmap, (width * px) / height, px, true);//http://javalove.egloos.com/m/67828
height = resized.getHeight();
width = resized.getWidth();
imageView.setImageBitmap(resized);
}
else imageView.setImageBitmap(bitmap);

Log.d("@@@", px + " " + width + " " + height );

}
});
dialog.show();
}

@Override
protected Void doInBackground(Void... voids) {

Detect();

return null;
}
}

@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);

// 상태바를 안보이도록 합니다.
getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
WindowManager.LayoutParams.FLAG_FULLSCREEN);

// 화면 켜진 상태를 유지합니다.
getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON,
WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

setContentView(R.layout.activity_main);

copyFile("shape_predictor_68_face_landmarks.dat");

mLayout = findViewById(R.id.layout_main);
surfaceView = findViewById(R.id.camera_preview_main);
mProgressBar = findViewById(R.id.progress1);

// 런타임 퍼미션 완료될때 까지 화면에서 보이지 않게 해야합니다.
surfaceView.setVisibility(View.GONE);

mProgressBar.setVisibility(View.GONE);

Button button1 = findViewById(R.id.button_main_capture);
button1.setOnClickListener(new View.OnClickListener() {

@Override
public void onClick(View v) {
mCameraPreview.takePicture();

}
});

Button button2 = findViewById(R.id.button_main_detection);
button2.setOnClickListener(new View.OnClickListener() {

@Override
public void onClick(View v) {

new Task().execute();
}
});

if (getPackageManager().hasSystemFeature(PackageManager.FEATURE_CAMERA)) {

int cameraPermission = ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA);
int writeExternalStoragePermission = ContextCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE);

if ( cameraPermission == PackageManager.PERMISSION_GRANTED
&& writeExternalStoragePermission == PackageManager.PERMISSION_GRANTED) {
startCamera();

}else {
if (ActivityCompat.shouldShowRequestPermissionRationale(this, REQUIRED_PERMISSIONS[0])
|| ActivityCompat.shouldShowRequestPermissionRationale(this, REQUIRED_PERMISSIONS[1])) {

Snackbar.make(mLayout, "이 앱을 실행하려면 카메라와 외부 저장소 접근 권한이 필요합니다.",
Snackbar.LENGTH_INDEFINITE).setAction("확인", new View.OnClickListener() {

@Override
public void onClick(View view) {

ActivityCompat.requestPermissions( MainActivity.this, REQUIRED_PERMISSIONS,
PERMISSIONS_REQUEST_CODE);
}
}).show();

} else {
// 2. 사용자가 퍼미션 거부를 한 적이 없는 경우에는 퍼미션 요청을 바로 합니다.
// 요청 결과는 onRequestPermissionResult에서 수신됩니다.
ActivityCompat.requestPermissions( this, REQUIRED_PERMISSIONS,
PERMISSIONS_REQUEST_CODE);
}

}

} else {

final Snackbar snackbar = Snackbar.make(mLayout, "디바이스가 카메라를 지원하지 않습니다.",
Snackbar.LENGTH_INDEFINITE);
snackbar.setAction("확인", new View.OnClickListener() {
@Override
public void onClick(View v) {
snackbar.dismiss();
}
});
snackbar.show();
}

}

void startCamera(){

// Create the Preview view and set it as the content of this Activity.
mCameraPreview = new CameraPreview(this, this, CAMERA_FACING, surfaceView);

}

@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grandResults) {

if ( requestCode == PERMISSIONS_REQUEST_CODE && grandResults.length == REQUIRED_PERMISSIONS.length) {

boolean check_result = true;

for (int result : grandResults) {
if (result != PackageManager.PERMISSION_GRANTED) {
check_result = false;
break;
}
}

if ( check_result ) {

startCamera();
}
else {

if (ActivityCompat.shouldShowRequestPermissionRationale(this, REQUIRED_PERMISSIONS[0])
|| ActivityCompat.shouldShowRequestPermissionRationale(this, REQUIRED_PERMISSIONS[1])) {

Snackbar.make(mLayout, "퍼미션이 거부되었습니다. 앱을 다시 실행하여 퍼미션을 허용해주세요. ",
Snackbar.LENGTH_INDEFINITE).setAction("확인", new View.OnClickListener() {

@Override
public void onClick(View view) {

finish();
}
}).show();

}else {

Snackbar.make(mLayout, "설정(앱 정보)에서 퍼미션을 허용해야 합니다. ",
Snackbar.LENGTH_INDEFINITE).setAction("확인", new View.OnClickListener() {

@Override
public void onClick(View view) {

finish();
}
}).show();
}
}

}

}
}

4-8. CameraPreview.java를 추가하여 다음 코드를 입력합니다.

package com.tistory.webnautes.dlibexample;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;

import android.content.Context;
import android.content.Intent;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Canvas;
import android.graphics.Matrix;
import android.hardware.Camera;
import android.hardware.Camera.Size;
import android.net.Uri;
import android.os.AsyncTask;
import android.os.Environment;
import androidx.appcompat.app.AppCompatActivity;
import android.util.Log;
import android.view.Surface;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.View;
import android.view.ViewGroup;

// 카메라에서 가져온 영상을 보여주는 카메라 프리뷰 클래스
class CameraPreview extends ViewGroup implements SurfaceHolder.Callback {

private final String TAG = "CameraPreview";

private int mCameraID;
private SurfaceView mSurfaceView;
private SurfaceHolder mHolder;
private Camera mCamera;
private Camera.CameraInfo mCameraInfo;
private int mDisplayOrientation;
private List<Size> mSupportedPreviewSizes;
private Size mPreviewSize;
private boolean isPreview = false;

private AppCompatActivity mActivity;

public CameraPreview(Context context, AppCompatActivity activity, int cameraID, SurfaceView surfaceView) {
super(context);

Log.d("@@@", "Preview");

mActivity = activity;
mCameraID = cameraID;
mSurfaceView = surfaceView;

mSurfaceView.setVisibility(View.VISIBLE);

// SurfaceHolder.Callback를 등록하여 surface의 생성 및 해제 시점을 감지
mHolder = mSurfaceView.getHolder();
mHolder.addCallback(this);

}

@Override
protected void onMeasure(int widthMeasureSpec, int heightMeasureSpec) {
// We purposely disregard child measurements because act as a
// wrapper to a SurfaceView that centers the camera preview instead
// of stretching it.
final int width = resolveSize(getSuggestedMinimumWidth(), widthMeasureSpec);
final int height = resolveSize(getSuggestedMinimumHeight(), heightMeasureSpec);
setMeasuredDimension(width, height);

if (mSupportedPreviewSizes != null) {
mPreviewSize = getOptimalPreviewSize(mSupportedPreviewSizes, width, height);
}
}

@Override
protected void onLayout(boolean changed, int l, int t, int r, int b) {
if (changed && getChildCount() > 0) {
final View child = getChildAt(0);

final int width = r - l;
final int height = b - t;

int previewWidth = width;
int previewHeight = height;
if (mPreviewSize != null) {
previewWidth = mPreviewSize.width;
previewHeight = mPreviewSize.height;
}

// Center the child SurfaceView within the parent.
if (width * previewHeight > height * previewWidth) {
final int scaledChildWidth = previewWidth * height / previewHeight;
child.layout((width - scaledChildWidth) / 2, 0,
(width + scaledChildWidth) / 2, height);
} else {
final int scaledChildHeight = previewHeight * width / previewWidth;
child.layout(0, (height - scaledChildHeight) / 2,
width, (height + scaledChildHeight) / 2);
}
}
}

// Surface가 생성되었을 때 어디에 화면에 프리뷰를 출력할지 알려줘야 한다.
public void surfaceCreated(SurfaceHolder holder) {

// Open an instance of the camera
try {
mCamera = Camera.open(mCameraID); // attempt to get a Camera instance
} catch (Exception e) {
// Camera is not available (in use or does not exist)
Log.e(TAG, "Camera " + mCameraID + " is not available: " + e.getMessage());
}

// retrieve camera's info.
Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
Camera.getCameraInfo(mCameraID, cameraInfo);

mCameraInfo = cameraInfo;
mDisplayOrientation = mActivity.getWindowManager().getDefaultDisplay().getRotation();

int orientation = calculatePreviewOrientation(mCameraInfo, mDisplayOrientation);
mCamera.setDisplayOrientation(orientation);

mSupportedPreviewSizes = mCamera.getParameters().getSupportedPreviewSizes();
requestLayout();

// get Camera parameters
Camera.Parameters params = mCamera.getParameters();

List<String> focusModes = params.getSupportedFocusModes();
if (focusModes.contains(Camera.Parameters.FOCUS_MODE_AUTO)) {
// set the focus mode
params.setFocusMode(Camera.Parameters.FOCUS_MODE_AUTO);
// set Camera parameters
mCamera.setParameters(params);
}

try {

mCamera.setPreviewDisplay(holder);

// Important: Call startPreview() to start updating the preview
// surface. Preview must be started before you can take a picture.
mCamera.startPreview();
isPreview = true;
Log.d(TAG, "Camera preview started.");
} catch (IOException e) {
Log.d(TAG, "Error setting camera preview: " + e.getMessage());
}

}

public void surfaceDestroyed(SurfaceHolder holder) {
// Surface will be destroyed when we return, so stop the preview.
// Release the camera for other applications.
if (mCamera != null) {
if (isPreview)
mCamera.stopPreview();
mCamera.release();
mCamera = null;
isPreview = false;
}

}

private Size getOptimalPreviewSize(List<Size> sizes, int w, int h) {
final double ASPECT_TOLERANCE = 0.1;
double targetRatio = (double) w / h;
if (sizes == null) return null;

Size optimalSize = null;
double minDiff = Double.MAX_VALUE;

int targetHeight = h;

// Try to find an size match aspect ratio and size
for (Size size : sizes) {
double ratio = (double) size.width / size.height;
if (Math.abs(ratio - targetRatio) > ASPECT_TOLERANCE) continue;
if (Math.abs(size.height - targetHeight) < minDiff) {
optimalSize = size;
minDiff = Math.abs(size.height - targetHeight);
}
}

// Cannot find the one match the aspect ratio, ignore the requirement
if (optimalSize == null) {
minDiff = Double.MAX_VALUE;
for (Size size : sizes) {
if (Math.abs(size.height - targetHeight) < minDiff) {
optimalSize = size;
minDiff = Math.abs(size.height - targetHeight);
}
}
}
return optimalSize;
}

public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {

// If your preview can change or rotate, take care of those events here.
// Make sure to stop the preview before resizing or reformatting it.

if (mHolder.getSurface() == null) {
// preview surface does not exist
Log.d(TAG, "Preview surface does not exist");
return;
}

// stop preview before making changes
try {
mCamera.stopPreview();
Log.d(TAG, "Preview stopped.");
} catch (Exception e) {
// ignore: tried to stop a non-existent preview
Log.d(TAG, "Error starting camera preview: " + e.getMessage());
}

int orientation = calculatePreviewOrientation(mCameraInfo, mDisplayOrientation);
mCamera.setDisplayOrientation(orientation);

try {
mCamera.setPreviewDisplay(mHolder);
mCamera.startPreview();
Log.d(TAG, "Camera preview started.");
} catch (Exception e) {
Log.d(TAG, "Error starting camera preview: " + e.getMessage());
}

}

/**
* 안드로이드 디바이스 방향에 맞는 카메라 프리뷰를 화면에 보여주기 위해 계산합니다.
*/
public static int calculatePreviewOrientation(Camera.CameraInfo info, int rotation) {
int degrees = 0;

switch (rotation) {
case Surface.ROTATION_0:
degrees = 0;
break;
case Surface.ROTATION_90:
degrees = 90;
break;
case Surface.ROTATION_180:
degrees = 180;
break;
case Surface.ROTATION_270:
degrees = 270;
break;
}

int result;
if (info.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
result = (info.orientation + degrees) % 360;
result = (360 - result) % 360; // compensate the mirror
} else { // back-facing
result = (info.orientation - degrees + 360) % 360;
}

return result;
}

public void takePicture(){

mCamera.takePicture(shutterCallback, rawCallback, jpegCallback);
}

Camera.ShutterCallback shutterCallback = new Camera.ShutterCallback() {
public void onShutter() {

}
};

Camera.PictureCallback rawCallback = new Camera.PictureCallback() {
public void onPictureTaken(byte[] data, Camera camera) {

}
};

//참고 : http://stackoverflow.com/q/37135675
Camera.PictureCallback jpegCallback = new Camera.PictureCallback() {
public void onPictureTaken(byte[] data, Camera camera) {

//이미지의 너비와 높이 결정
int w = camera.getParameters().getPictureSize().width;
int h = camera.getParameters().getPictureSize().height;
int orientation = calculatePreviewOrientation(mCameraInfo, mDisplayOrientation);

//byte array를 bitmap으로 변환
BitmapFactory.Options options = new BitmapFactory.Options();
options.inPreferredConfig = Bitmap.Config.ARGB_8888;
Bitmap bitmap = BitmapFactory.decodeByteArray( data, 0, data.length, options);

//이미지를 디바이스 방향으로 회전
Matrix matrix = new Matrix();
matrix.postRotate(orientation);
bitmap = Bitmap.createBitmap(bitmap, 0, 0, w, h, matrix, true);

//bitmap을 byte array로 변환
//ByteArrayOutputStream stream = new ByteArrayOutputStream();
//bitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream);
//bitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream);
//byte[] currentData = stream.toByteArray();

//파일로 저장
//new SaveImageTask().execute(currentData);
new SaveImageTask().execute(FlipVertically(bitmap));

}
};

private class SaveImageTask extends AsyncTask<Bitmap, Void, Void> {
String fullpath;

@Override
protected void onPostExecute(Void aVoid) {
super.onPostExecute(aVoid);

mCamera.startPreview();

// 갤러리에 반영
File a = new File(fullpath);
Intent mediaScanIntent = new Intent( Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
mediaScanIntent.setData(Uri.fromFile(a));
getContext().sendBroadcast(mediaScanIntent);

try {
mCamera.setPreviewDisplay(mHolder);
mCamera.startPreview();
Log.d(TAG, "Camera preview started.");
} catch (Exception e) {
Log.d(TAG, "Error starting camera preview: " + e.getMessage());
}
}

@Override
protected Void doInBackground(Bitmap... data) {
FileOutputStream outStream = null;

try {

File path = new File (Environment.getExternalStorageDirectory().getAbsolutePath() + "/camtest/");
if (!path.exists()) {
path.mkdirs();
}

String filepath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/camtest/" + "1.bmp";
save(data[0], filepath);

fullpath = filepath;

// String fileName = String.format("%d.jpg", System.currentTimeMillis());
// File outputFile = new File(path, "1.jpg");
//
// outStream = new FileOutputStream(outputFile);
// outStream.write(data[0]);
// outStream.flush();
// outStream.close();

Log.d(TAG, "onPictureTaken - wrote bytes: " + data.length + " to "
+ path);

} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

return null;
}

}

// whats-online.info/science-and-tutorials/134/Android-code-How-to-Flip-an-Image-vertically-programmatically/
public Bitmap FlipVertically(Bitmap originalImage) {
// The gap we want between the flipped image and the original image
final int flipGap = 4;

int width = originalImage.getWidth();
int height = originalImage.getHeight();

// This will not scale but will flip on the Y axis
Matrix matrix = new Matrix();
matrix.preScale(1, -1);

// Create a Bitmap with the flip matrix applied to it.
// We only want the bottom half of the image
Bitmap flipImage = Bitmap.createBitmap(originalImage, 0,0 , width, height, matrix, true);

// Create a new bitmap with same width but taller to fit reflection
Bitmap bitmapWithFlip = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);

// Create a new Canvas with the bitmap that's big enough for
Canvas canvas = new Canvas(bitmapWithFlip);
//
// //Draw original image
// canvas.drawBitmap(originalImage, 0, height+flipGap, null);

//Draw the Flipped Image
canvas.drawBitmap(flipImage, 0, 0, null);

return bitmapWithFlip;
}

//https://stackoverflow.com/a/22914268
private static final int BMP_WIDTH_OF_TIMES = 4;
private static final int BYTE_PER_PIXEL = 3;

/**
* Android Bitmap Object to Window's v3 24bit Bmp Format File
* @param orgBitmap
* @param filePath
* @return file saved result
*/
public static boolean save(Bitmap orgBitmap, String filePath) throws IOException {
long start = System.currentTimeMillis();
if(orgBitmap == null){
return false;
}

if(filePath == null){
return false;
}

boolean isSaveSuccess = true;

//image size
int width = orgBitmap.getWidth();
int height = orgBitmap.getHeight();

//image dummy data size
//reason : the amount of bytes per image row must be a multiple of 4 (requirements of bmp format)
byte[] dummyBytesPerRow = null;
boolean hasDummy = false;
int rowWidthInBytes = BYTE_PER_PIXEL * width; //source image width * number of bytes to encode one pixel.
if(rowWidthInBytes%BMP_WIDTH_OF_TIMES>0){
hasDummy=true;
//the number of dummy bytes we need to add on each row
dummyBytesPerRow = new byte[(BMP_WIDTH_OF_TIMES-(rowWidthInBytes%BMP_WIDTH_OF_TIMES))];
//just fill an array with the dummy bytes we need to append at the end of each row
for(int i = 0; i < dummyBytesPerRow.length; i++){
dummyBytesPerRow[i] = (byte)0xFF;
}
}

//an array to receive the pixels from the source image
int[] pixels = new int[width * height];

//the number of bytes used in the file to store raw image data (excluding file headers)
int imageSize = (rowWidthInBytes+(hasDummy?dummyBytesPerRow.length:0)) * height;
//file headers size
int imageDataOffset = 0x36;

//final size of the file
int fileSize = imageSize + imageDataOffset;

//Android Bitmap Image Data
orgBitmap.getPixels(pixels, 0, width, 0, 0, width, height);

//ByteArrayOutputStream baos = new ByteArrayOutputStream(fileSize);
ByteBuffer buffer = ByteBuffer.allocate(fileSize);

/**
* BITMAP FILE HEADER Write Start
**/
buffer.put((byte)0x42);
buffer.put((byte)0x4D);

//size
buffer.put(writeInt(fileSize));

//reserved
buffer.put(writeShort((short)0));
buffer.put(writeShort((short)0));

//image data start offset
buffer.put(writeInt(imageDataOffset));

/** BITMAP FILE HEADER Write End */

//*******************************************

/** BITMAP INFO HEADER Write Start */
//size
buffer.put(writeInt(0x28));

//width, height
//if we add 3 dummy bytes per row : it means we add a pixel (and the image width is modified.
buffer.put(writeInt(width+(hasDummy?(dummyBytesPerRow.length==3?1:0):0)));
buffer.put(writeInt(height));

//planes
buffer.put(writeShort((short)1));

//bit count
buffer.put(writeShort((short)24));

//bit compression
buffer.put(writeInt(0));

//image data size
buffer.put(writeInt(imageSize));

//horizontal resolution in pixels per meter
buffer.put(writeInt(0));

//vertical resolution in pixels per meter (unreliable)
buffer.put(writeInt(0));

buffer.put(writeInt(0));

buffer.put(writeInt(0));

/** BITMAP INFO HEADER Write End */

int row = height;
int col = width;
int startPosition = (row - 1) * col;
int endPosition = row * col;
while( row > 0 ){
for(int i = startPosition; i < endPosition; i++ ){
buffer.put((byte)(pixels[i] & 0x000000FF));
buffer.put((byte)((pixels[i] & 0x0000FF00) >> 8));
buffer.put((byte)((pixels[i] & 0x00FF0000) >> 16));
}
if(hasDummy){
buffer.put(dummyBytesPerRow);
}
row--;
endPosition = startPosition;
startPosition = startPosition - col;
}

FileOutputStream fos = new FileOutputStream(filePath);
fos.write(buffer.array());
fos.close();
Log.v("AndroidBmpUtil" ,System.currentTimeMillis()-start+" ms");

return isSaveSuccess;
}

/**
* Write integer to little-endian
* @param value
* @return
* @throws IOException
*/
private static byte[] writeInt(int value) throws IOException {
byte[] b = new byte[4];

b[0] = (byte)(value & 0x000000FF);
b[1] = (byte)((value & 0x0000FF00) >> 8);
b[2] = (byte)((value & 0x00FF0000) >> 16);
b[3] = (byte)((value & 0xFF000000) >> 24);

return b;
}

/**
* Write short to little-endian byte array
* @param value
* @return
* @throws IOException
*/
private static byte[] writeShort(short value) throws IOException {
byte[] b = new byte[2];

b[0] = (byte)(value & 0x00FF);
b[1] = (byte)((value & 0xFF00) >> 8);

return b;
}
}

4-9. MainActivity.java의 Detect를 클릭하면 왼쪽에 보이는 빨간 전구를 클릭합니다.

메뉴에서 Create JNI function을 선택합니다.

native-lib.cpp에 함수가 추가됩니다.

4-10. native-lib.cpp 파일을 다음처럼 수정합니다.

기존 코드에서 필요없는 부분을 주석처리하고 추가로 코드를 복사해주면됩니다.

dlib 라이브러리가 아직 인식안되어 해당 코드들이 빨간색으로 보입니다. 다음 단계를 진행하고 나면 해결됩니다.

#include <jni.h>
//#include <string>
#include <dlib/image_processing/frontal_face_detector.h>
#include <dlib/image_processing/render_face_detections.h>
#include <dlib/image_processing.h>
#include <dlib/image_transforms.h>
#include <dlib/image_io.h>
#include <iostream>
#include <android/log.h>

using namespace dlib;
using namespace std;

//extern "C" JNIEXPORT jstring JNICALL
//Java_com_tistory_webnautes_dlibexample_MainActivity_stringFromJNI(
// JNIEnv* env,
// jobject /* this */) {
// std::string hello = "Hello from C++";
// return env->NewStringUTF(hello.c_str());
//}
extern "C"
JNIEXPORT void JNICALL
Java_com_tistory_webnautes_dlibexample_MainActivity_Detect(JNIEnv *env, jobject instance) {

__android_log_print(ANDROID_LOG_DEBUG, "native-lib :: ",
"start");

try {
// We need a face detector. We will use this to get bounding boxes for
// each face in an image.
frontal_face_detector detector = get_frontal_face_detector();

// And we also need a shape_predictor. This is the tool that will predict face
// landmark positions given an image and face bounding box. Here we are just
// loading the model from the shape_predictor_68_face_landmarks.dat file you gave
// as a command line argument.
shape_predictor sp;
deserialize("/storage/emulated/0/shape_predictor_68_face_landmarks.dat") >> sp;

//image_window win, win_faces;
// Loop over all the images provided on the command line.
__android_log_print(ANDROID_LOG_DEBUG, "native-lib :: ",
"load shape_predictor_68_face_landmarks");

array2d<rgb_pixel> img;
load_image(img, "/storage/emulated/0/camtest/1.bmp");

// Make the image larger so we can detect small faces.
// pyramid_up(img);

__android_log_print(ANDROID_LOG_DEBUG, "native-lib :: ",
"load image %d x %d");

array2d<rgb_pixel> sizeImg(1280, 960);
resize_image(img, sizeImg);
;

__android_log_print(ANDROID_LOG_DEBUG, "native-lib :: ",
"start detect");

// Now tell the face detector to give us a list of bounding boxes
// around all the faces in the image.
std::vector<rectangle> dets = detector(sizeImg, 0);
__android_log_print(ANDROID_LOG_DEBUG, "native-lib :: ",
"Number of faces detected: %d", (int) dets.size());

for ( auto & rect : dets)
dlib :: draw_rectangle (sizeImg, rect, dlib :: rgb_pixel ( 255 , 0 , 0 ), 3 );

// Now we will go ask the shape_predictor to tell us the pose of
// each face we detected.
std::vector<full_object_detection> shapes;
for (unsigned long j = 0; j < dets.size(); ++j)
{
full_object_detection shape = sp(img, dets[j]);
cout << "number of parts: "<< shape.num_parts() << endl;

for( int i=0; i<shape.num_parts(); i++){

point p = shape.part(i);

dlib :: draw_solid_circle(sizeImg, p, 3, dlib :: rgb_pixel ( 255 , 0 , 0 ));
}
}

dlib :: save_bmp (sizeImg, "/storage/emulated/0/camtest/output.bmp" );

} catch (exception& e)
{

__android_log_print(ANDROID_LOG_DEBUG, "native-lib :: ",
"exception thrown! %s", e.what() );
}

}

4-11. CMakeLists.txt 파일을 수정하여 빌드시 DLib를 사용하도록 합니다.

pathPROJECT를 현재 프로젝트 경로로 변경하세요.

안드로이드 스튜디오 메뉴에서 File > Sync Project with Gradle Files를 선택합니다.

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.4.1)

set(CMAKE_VERBOSE_MAKEFILE on)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")

set(pathPROJECT C:/Users/webnautes/AndroidStudioProjects/DLibExample)
set(DLIB_PATH ${pathPROJECT}/dlib-native)
add_library(dlib SHARED IMPORTED)

set_target_properties( dlib
        PROPERTIES IMPORTED_LOCATION
        ${pathPROJECT}/app/src/main/JniLibs/arm64-v8a/libdlib.so )

include_directories(${DLIB_PATH}/include)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
        native-lib

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        ${pathPROJECT}/app/src/main/cpp/native-lib.cpp )

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log )
find_library( android-lib android)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        native-lib

        dlib

        ${android-lib}

        # Links the target library to the log library
        # included in the NDK.
        ${log-lib} )

4-12. AndroidManifest.xml에 사용할 권한을 추가합니다.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.tistory.webnautes.dlibexample">

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

<application
android:allowBackup="true"

API 29 이상에서는 다음 옵션을 추가해야 합니다.

<application
android:requestLegacyExternalStorage="true"

4-13. 아래 경로에 있는 gui_core_kernel_2.h 파일 상단의 다음 두줄을 주석처리합니다.

DLIB_NO_GUI_SUPPORT 관련 오류를 강제로 나지 않도록합니다.

파란색 부분은 안드로이드 프로젝트 경로입니다.

C:\Users\webnautes\AndroidStudioProjects\DlibExample\dlib-native\include\dlib\gui_core\gui_core_kernel_2.h

#ifdef DLIB_NO_GUI_SUPPORT
//#error "DLIB_NO_GUI_SUPPORT is defined so you can't use the GUI code. Turn DLIB_NO_GUI_SUPPORT off if you want to use it."
//#error "Also make sure you have libx11-dev installed on your system"
#endif

4-14. 프로젝트 창의 main 아래에 JniLibs 디렉토리를 생성 후, 다시 JniLibs 디렉토리에 arm64-v8a 디렉토리를 생성합니다.

그리고나서 Ctrl키를 누른채 C:\dlib-native\lib에 있는 libdlib.so 파일을 드래그해서 복사해줍니다.

앱 설치후, 라이브러리 파일을 못찾는 현상이 발견되서 추가했습니다.