The following lines are the errors:

running build
running build_py
running egg_info
writing fcos.egg-info/PKG-INFO
writing dependency_links to fcos.egg-info/dependency_links.txt
writing requirements to fcos.egg-info/requires.txt
writing top-level names to fcos.egg-info/top_level.txt
reading manifest file 'fcos.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no directories found matching 'fcos/configs'
writing manifest file 'fcos.egg-info/SOURCES.txt'
running build_ext
building 'fcos_core.C' extension
Emitting ninja build file /data1/FCOS/build/temp.linux-x86_64-3.6/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] /usr/local/cuda/bin/nvcc -DWITH_CUDA -Ifcos_core/csrc -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/TH -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/local/miniconda3/include/python3.6m -c -c /data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu -o /data1/FCOS/build/temp.linux-x86_64-3.6/fcos_core/csrc/cuda/deform_conv_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_70,code=sm_70 -std=c++14
FAILED: /data1/FCOS/build/temp.linux-x86_64-3.6/fcos_core/csrc/cuda/deform_conv_cuda.o
/usr/local/cuda/bin/nvcc -DWITH_CUDA -Ifcos_core/csrc -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/TH -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/local/miniconda3/include/python3.6m -c -c /data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu -o /data1/FCOS/build/temp.linux-x86_64-3.6/fcos_core/csrc/cuda/deform_conv_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_70,code=sm_70 -std=c++14
/data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu(72): error: identifier "AT_CHECK" is undefined

/data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu(200): error: identifier "AT_CHECK" is undefined

/data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu(307): error: identifier "AT_CHECK" is undefined

/data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu(423): error: identifier "AT_CHECK" is undefined

/data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu(504): error: identifier "AT_CHECK" is undefined

/data1/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu(586): error: identifier "AT_CHECK" is undefined

6 errors detected in the compilation of "/tmp/tmpxft_00000aad_00000000-6_deform_conv_cuda.cpp1.ii".
[2/2] /usr/local/cuda/bin/nvcc -DWITH_CUDA -Ifcos_core/csrc -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/TH -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/local/miniconda3/include/python3.6m -c -c /data1/FCOS/fcos_core/csrc/cuda/deform_pool_cuda.cu -o /data1/FCOS/build/temp.linux-x86_64-3.6/fcos_core/csrc/cuda/deform_pool_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_70,code=sm_70 -std=c++14
FAILED: /data1/FCOS/build/temp.linux-x86_64-3.6/fcos_core/csrc/cuda/deform_pool_cuda.o
/usr/local/cuda/bin/nvcc -DWITH_CUDA -Ifcos_core/csrc -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/TH -I/usr/local/miniconda3/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/local/miniconda3/include/python3.6m -c -c /data1/FCOS/fcos_core/csrc/cuda/deform_pool_cuda.cu -o /data1/FCOS/build/temp.linux-x86_64-3.6/fcos_core/csrc/cuda/deform_pool_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_70,code=sm_70 -std=c++14
/data1/FCOS/fcos_core/csrc/cuda/deform_pool_cuda.cu(42): error: identifier "AT_CHECK" is undefined

/data1/FCOS/fcos_core/csrc/cuda/deform_pool_cuda.cu(68): error: identifier "AT_CHECK" is undefined

2 errors detected in the compilation of "/tmp/tmpxft_00000aae_00000000-6_deform_pool_cuda.cpp1.ii".
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/usr/local/miniconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1400, in _run_ninja_build
check=True)
File "/usr/local/miniconda3/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "setup.py", line 77, in
include_package_data=True,
File "/usr/local/miniconda3/lib/python3.6/site-packages/setuptools/init.py", line 144, in setup
return distutils.core.setup(**attrs)
File "/usr/local/miniconda3/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/miniconda3/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/local/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/miniconda3/lib/python3.6/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/local/miniconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/miniconda3/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 87, in run
_build_ext.run(self)
File "/usr/local/miniconda3/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/usr/local/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/local/miniconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 580, in build_extensions
build_ext.build_extensions(self)
File "/usr/local/miniconda3/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/usr/local/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/usr/local/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/usr/local/miniconda3/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "/usr/local/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
depends=ext.depends)
File "/usr/local/miniconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 423, in unix_wrap_ninja_compile
with_cuda=with_cuda)
File "/usr/local/miniconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "/usr/local/miniconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1413, in _run_ninja_build
raise RuntimeError(message)

Solution: replaced "ATT_CHECK" to "TORCH_CHECK" in file "/FCOS/fcos_core/csrc/cuda/deform_conv_cuda.cu" and "/FCOS/fcos_core/csrc/cuda/deform_pool_cuda.cu"