Openresty realizes picture (file) server

Openresty realizes picture (file) server

Click to open the link

Introduction

prologue

This function is a picture (file) saving function implemented by openresty's lua script, and the file upload is developed using java code

Data definition

Upload data and file information do not distinguish between before and after, but the system will only save the last pair of information

  • Data Format:
{"fileDir":"The directory where the file is saved","fileName":"File name"} Copy code
  • 1
  • Return result
{"status":"Whether it was successful","result":"Return result","msg":"Exception reason"} enum status:["success","failed"] Copy code
  • \
  • The folder
    where the save folder  is saved is defined in the perfix variable of nginx

Code

Nginx configuration

as follows:

server { listen 80; server_name localhost; # Configure the saved folder set $prefix "/data"; location/uploadimage { # Whether the configuration takes effect every time lua changes, suitable for debugging # lua_code_cache off; # Configure lua script content_by_lua_file/openresty-web/luascript/luascript; } # Used to cooperate to understand the structure of the incoming message to nginx location/uploadtest{ # lua_code_cache off; content_by_lua_file/openresty-web/luascript/luauploadtest; } error_page 500 502 503 504/50x.html; location =/50x.html { root html; } } Copy code
  • \

lua script

luascript:

package.path ='/openresty-web/lualib/resty/?.lua;' local upload = require "upload" local cjson = require("cjson") Result={status="success",result="",msg=""} Result.__index=Result function Result.conSuccess(ret) ret["status"]="success" ret["result"]="upload success" return ret end function Result.conFailed(ret,err) ret["status"]="failed" ret["msg"]=err ret["result"]="upload failed" return ret end function Result:new() local ret={} setmetatable({},Result) return ret end - lua-resty-upload local chunk_size = 4096 local form = upload:new(chunk_size) if not form then ngx.say(cjson.encode(Result.conFailed(Result:new(),"plase upload right info"))) return end local file local filelen=0 form:set_timeout(0) - 1 sec local filename local prefix=ngx.var.prefix - Match the file name, the current case is used to determine whether it is a file module function get_filename(res) local filename = ngx.re.match(res,'(.+)filename="(.+)"(.*)') if filename then return filename[2] end end - Used to open the input stream, automatically created when the folder does not exist function openstream(fileinfo,opt) local file,err=io.open(prefix..fileinfo["fileDir"],"r") if not file then local start=string.find(err,"No such file or directory") if start then local exeret=os.execute("mkdir -p "..prefix..fileinfo["fileDir"]) if exeret ~= 0 then return nil,"Make directory failed" end else return nil,err end end file,err=io.open(prefix..fileinfo["fileDir"]..fileinfo["fileName"],opt) return file,err end local osfilepath local tmpfiletbl local hasFile=false local loopfile=false local fileinfostr local fileinfo local result=Result:new() - Read files and file information cyclically while true do local typ, res, err = form:read() if not typ then break end if typ == "header" then if res[1] ~= "Content-Type" then filename = get_filename(res[2]) if filename then loopfile=true hasFile=true - Determine whether there is file information - If there is no recording memory if fileinfo then file,err=openstream(fileinfo,"w") if not file then break end else tmpfiletbl={} end else loopfile = false fileinfostr = "" end end end if loopfile then if typ == "body" then if file then filelen = filelen + tonumber(string.len(res)) file:write(res) else table.insert(tmpfiletbl,res) end elseif typ == "part_end" then if file then file:close() file = nil end end else if typ == "body" then fileinfostr=fileinfostr .. res elseif typ == "part_end" then fileinfo = cjson.decode(fileinfostr) end end if typ == "eof" then break end end if not hasFile then err="plase upload file" elseif not fileinfo or not fileinfo["fileDir"] or not fileinfo["fileName"] then err="plase offer file info" end if err then ngx.log(ngx.ERR,err) Result.conFailed(result,err) ngx.say(cjson.encode(result)) return end - Because there is file information transmitted after the file - So you need to print the file information input to the memory to the disk if tmpfiletbl and table.getn(tmpfiletbl)> 0 then file,err=openstream(fileinfo,"w") if not file then ngx.log(ngx.ERR,err) Result.conFailed(result,err) ngx.say(cjson.encode(result)) return else for index,value in ipairs(tmpfiletbl) do filelen = filelen + tonumber(string.len(value)) file:write(value) end file:close() file=nil end end Result.conSuccess(result) ngx.say(cjson.encode(result)) Copy code
  • \

luauploadtest:

local upload = require "resty.upload" local cjson = require "cjson" local chunk_size = 5 - should be set to 4096 or 8192 - for real-world settings local form, err = upload:new(chunk_size) if not form then ngx.log(ngx.ERR, "failed to new upload: ", err) ngx.exit(500) end form:set_timeout(1000) - 1 sec while true do local typ, res, err = form:read() if not typ then ngx.say("failed to read: ", err) return end ngx.say("read: ", cjson.encode({typ, res})) if typ == "eof" then break end end local typ, res, err = form:read() ngx.say("read: ", cjson.encode({typ, res})) Copy code
  • \

The luauploadtest code is the official code

Java

ImageServer:

package cn.com.cgbchina.image; import cn.com.cgbchina.image.exception.ImageDeleteException; import cn.com.cgbchina.image.exception.ImageUploadException; import org.springframework.web.multipart.MultipartFile; /** * Created by 11140721050130 on 16-3-22. */ public interface ImageServer { /** * Delete Files * * @param fileName file name * @return is the deletion successful */ boolean delete(String fileName) throws ImageDeleteException; /** * * @param originalName original file name * @param file file * @return relative path after file upload */ String upload(String originalName, MultipartFile file) throws ImageUploadException; } Copy code
  • \

LuaResult:

package cn.com.cgbchina.image.nginx; import lombok.Getter; import lombok.Setter; /** * Comment: Used to save the returned result, * Originally wanted to be put into the inner class of LuaImageServiceImpl, * But Jackson does not support it and cannot be deserialized * Created by ldaokun2006 on 2017/10/24. */ @Setter @Getter public class LuaResult{ private LuaResultStatus status; private String result; private String msg; private String httpUrl; public LuaResult(){} public void setStatus(String result){ status=LuaResultStatus.valueOf(result.toUpperCase()); } public enum LuaResultStatus{ SUCCESS, FAILED; } } Copy code
  • \

ImageServerImpl:

package cn.com.cgbchina.image.nginx; import cn.com.cgbchina.common.utils.DateHelper; import cn.com.cgbchina.image.ImageServer; import cn.com.cgbchina.image.exception.ImageDeleteException; import cn.com.cgbchina.image.exception.ImageUploadException; import com.github.kevinsawicki.http.HttpRequest; import com.google.common.base.Splitter; import com.spirit.util.JsonMapper; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.Setter; import lombok.extern.slf4j.Slf4j; import org.springframework.stereotype.Service; import org.springframework.web.multipart.MultipartFile; import java.io.*; import java.util.ArrayList; import java.util.Date; import java.util.List; import java.util.UUID; import java.util.concurrent.*; import java.util.concurrent.atomic.AtomicInteger; /** * Comment: realize file upload function * Created by ldaokun2006 on 2017/10/16. */ @Service @Slf4j public class LuaImageServiceImpl implements ImageServer{ //For storing nginx server url, some architectures will have multiple places for pictures private List<String> httpUrls; private ExecutorService fixedThreadPool; private Integer timeout; private int threadSize=50; public LuaImageServiceImpl(String httpUrls){ this(httpUrls,30000); } /** * * @param httpUrls stores the nginx server url * @param timeout http timeout */ public LuaImageServiceImpl(String httpUrls,int timeout){ this.httpUrls=Splitter.on(";").splitToList(httpUrls); //Nothing to see, just want to make the name of the thread pool easier to understand this.fixedThreadPool = new ThreadPoolExecutor(threadSize, threadSize, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(),new ThreadFactory(){ private final AtomicInteger poolNumber = new AtomicInteger(1); private final ThreadGroup group; private final AtomicInteger threadNumber = new AtomicInteger(1); private final String namePrefix; { SecurityManager s = System.getSecurityManager(); group = (s != null)? s.getThreadGroup(): Thread.currentThread().getThreadGroup(); namePrefix = "LuaUploadPool-" + poolNumber.getAndIncrement() + "-thread-"; } public Thread newThread(Runnable r) { Thread t = new Thread(group, r, namePrefix + threadNumber.getAndIncrement(), 0); if (t.isDaemon()) t.setDaemon(false); if (t.getPriority() != Thread.NORM_PRIORITY) t.setPriority(Thread.NORM_PRIORITY); return t; } }); this.timeout=timeout; } /** * Comment: No need to develop delete function * @param fileName file name * @return * @throws ImageDeleteException */ @Override public boolean delete(String fileName) throws ImageDeleteException { return true; } /** * Commont: Used for SpringMVC * @param originalName original file name * @param file file * @return * @throws ImageUploadException */ @Override public String upload(String originalName, MultipartFile file) throws ImageUploadException { try { return this.upload(originalName,file.getInputStream()); } catch (IOException e) { log.error("upload fail: "+ e.getMessage(), e); throw new ImageUploadException("upload fail: "+e.getMessage(),e); } } /** * Commont: Upload image core code * @param originalName original file name * @param inputStream The file stream of the file to be uploaded * @return * @throws ImageUploadException */ private String upload(String originalName,InputStream inputStream) throws ImageUploadException { ByteArrayOutputStream byteOutStream = null; try { //Prepare data byte[] tmpData=new byte[1024]; byte[] inputData; byteOutStream = new ByteArrayOutputStream(); int len=0; while((len=inputStream.read(tmpData,0,tmpData.length))!=-1){ byteOutStream.write(tmpData,0,len); } inputData=byteOutStream.toByteArray(); LuaSend sendInfo = new LuaSend(generateFileDir(),generateFileName(originalName)); List<Future<LuaResult>> resultList=new ArrayList<>(httpUrls.size()); //send pictures for(String httpUrl:httpUrls) { SendImg sendImg = new SendImg(httpUrl,sendInfo, inputData,this.timeout); resultList.add(fixedThreadPool.submit(sendImg)); } for(Future<LuaResult> future:resultList) { //Thread pool exception is thrown here LuaResult resultLuaResult = future.get(); if (LuaResult.LuaResultStatus.SUCCESS != resultLuaResult.getStatus()) { throw new ImageUploadException("lua result url:"+resultLuaResult.getHttpUrl()+" msg: "+ resultLuaResult.getMsg()); } } return sendInfo.toString(); }catch (Exception e){ log.error("upload fail: "+e.getMessage(),e); throw new ImageUploadException("upload fail: "+e.getMessage(),e); }finally { try { if(byteOutStream!=null) { byteOutStream.close(); } if(inputStream!=null) { inputStream.close(); } } catch (IOException e) { throw new ImageUploadException("upload fail: "+e.getMessage(),e); } } } String separator=File.separator; String dateFormat=separator+"yyyy"+separator+"MM"+separator+"dd"+ separator; /** * Comment: Make the path according to the time to prevent too much stuff in a certain folder * @return returns the path to be saved */ private String generateFileDir(){ return DateHelper.date2string(new Date(),dateFormat); } /** * Comment: Use UUID to prevent duplication of file names * @param originalName source file name * @return the name of the file to be saved */ private String generateFileName(String originalName){ return UUID.randomUUID().toString(); } /** * Comment: Used to send pictures */ @AllArgsConstructor class SendImg implements Callable<LuaResult>{ private String httpUrl; private LuaSend sendInfo; private byte[] inputStream; private Integer timeout; @Override public LuaResult call() throws Exception { try { String resultStr = HttpRequest .post(httpUrl, false) .part("fileInfo", JsonMapper.JSON_NON_EMPTY_MAPPER.toJson(sendInfo)) //There is a pit in this place, part must use this method to upload pictures, //Cannot use without Content-Type and fileName .part("file", sendInfo.getFileName(), "multipart/form-data; boundary=00content0boundary00", new ByteArrayInputStream(inputStream)) .connectTimeout(timeout).body(); log.info("result:"+resultStr); LuaResult result = JsonMapper.JSON_NON_DEFAULT_MAPPER.fromJson(resultStr, LuaResult.class); result.setHttpUrl(httpUrl); return result; }catch(Exception e){ throw new ImageUploadException("upload failed url:"+httpUrl+" info:"+sendInfo.toString(),e); } } } /** * Comment: file data */ @Setter @Getter @AllArgsConstructor class LuaSend { //File Directory private String fileDir; //file name private String fileName; @Override public String toString(){ return fileDir+fileName; } } /** * Comment: For testing * @param args * @throws ImageUploadException * @throws FileNotFoundException */ public static void main(String[] args) throws ImageUploadException, FileNotFoundException { LuaImageServiceImpl service=new LuaImageServiceImpl("http://192.168.99.102/uploadimage"); try { System.out.println(service.upload("qqqqq", new FileInputStream("D:\\shsh.txt"))); }finally { service.fixedThreadPool.shutdown(); } } } Copy code
  • \

summary

possible problems

  1. When uploading two pictures or picture information, the system only keeps the last information
  2. The picture and picture information can be placed at will, but the two must be sent in pairs. It is recommended to send the picture information first and then send the picture, so that the picture does not need to be saved in the memory at lua
  3. When uploading a large picture, there will be a prompt that the file is too large, which needs to be added in the nginx configuration file
    client_max_body_size 100M;
  4. The Content-Type of Http Header must be used
    multipart/form-data; boundary=00content0boundary00
    , Boundary must exist, otherwise it is not easy
  5. To send pictures HttpRequest.part upload pictures must indicate the Content-type and fileName, otherwise it is not easy to use, but the Content-type does not need to be used in an example way
  6. Picture information must be copied into byte type, because it needs to be sent separately when using multiple threads

Problems encountered in development

  1. Send pictures HttpRequest.part Upload pictures must indicate the Content-type, otherwise it is not easy to use
  2. Jackson and fastjson must have parameterless constructors for classes that need to be deserialized, and they cannot be inner classes
  3. If lua's string.find is not found, the return result is
    nil
  4. CSDN editor, no need to function not easy to use

Involves knowledge

  1. HttpRequest.part is used to upload
    Content-type:multipart/form-data;
  2. Use of lua: www.runoob.com/lua/lua-tut...
  3. openresty's api: openresty.org/cn/componen...

\